The reconquista. When Christian Spanish kingdoms were conquering Muslim Iberia they took part of North Africa with it, as a treat, and just kept it since. The (honestly kinda sensible) Spanish position is these are core Spanish territories that simply happen to be in Africa rather than colonial territories that should be given back. Historically the Spain (or equivalents)-Morocco (or equivalents) border has been pretty fluid, so there’s no reason per se that “Africans stay in Africa, Europeans stay in Europe” should hold here, but in that case they don’t get to complain about having a land border with their former colonies.