Fears for economy as number of EU migrants leaving Brexit Britain surges