Where Did Conservatives Get the Idea that Immigrants Are a Fifth Column?

They got it from liberals, of course. For example, liberalrob (a commenter on this post of Megan McArdle):

Leaving aside the estimated 12 million poor who successfully came over our impervious borders at the current moment (that'd be a pretty respectable army if some bright boy gave them all guns, wouldn't it?), let me borrow a phrase from our current Vice President and say that that is pre-9/11 thinking. No, the angry poor don't currently have a massive armada of ships poised to invade our shores. That doesn't mean they might not get one. Look how close Osama is to getting his own atomic bomb; if things fall his way in Pakistan, he just might.

………

And all it takes, really, is the right guy (or gal) to come along and organize those people, equip them minimally, whip them into a frenzy of righteous indignation at their mistreatment, and point them in the right direction. Maybe it sounds preposterous to you that that keeps John Bowe awake at night. It has something of a racist "fear of the brown people" feel to it, I agree; but it also has a kernel of truth to it. Desperate people are susceptible to demagoguery, and few are more desperate than these exploited workers.

On the other hand, it might make more sense to call their bluff. It was a bluff when they warned of a “long hot summer” full of urban riots if Reagan were elected. It was a bluff when they threatened to make the country ungovernable if Bush were re-elected. It was a bluff when they claimed the Israeli crackdown on Palestinians would intensify the terrorism.

The “root cause” of fifth-column activity is the belief that the Establishment will back down in response to intimidation. If it doesn't the fifth column will disappear.