I believe part of the answer lies in the original folklore views on magic.
Before and during the medieval era, according to the Christian Church, magic was a foul, demon-granted perversion of the natural order. The people who practised magic were seen as foul, diseased old hags and wicked sorcerers who used the blood of Christian babies and dealt with Satan himself. Those accused of witchcraft were not the sorts of people likely to be in a lot of battles, sword vs. sword. Magic was seen as a weak and womanly (which back then meant cowardly) weapon, and was linked to disease, poison and other modern-day 'debuffs'.
These wizards were therefore not seen as being the sorts of people physically capable of fighting in an army. Scholars were also, I believe, seen as weak men. Therefore, there is a longstanding tradition of wizards not using armour as they wouldn't fight openly. Another factor was that many people believed that witches could, for example, harden their skin to avoid swordblows. Therefore, they wouldn't NEED armour anyway.
Renan Malke Stigliani also raised an excellent point about Merlin, a very famous wizard, who was powerful and wise but too old to fight with a sword and armour, again reinforcing magic as an alternative to physical combat, as opposed to a complementary tactic.
Then we get to the most famous wizard of all, Gandalf. He was a mystic, nearly all-powerful figure. He had no need of armour, as he was never injured. His power was in other things than physical combat.
Other ideas (in other cultures) for how magic worked involved special clothes, jewellery, motions, sigils, etc. and were not really portable. A Druid might cast a good luck charm to make the crops grow, but he wouldn't dance about in full armour. Often, magic then had much less to do with fighting than the pop-culture idea now, so wizards again had no need to be uncomfortably dressed in solid metal plates while casting spells.
Then, when D&D was created, along with the creation of the concept of a weak wizard going out and fighting mosters in dungeons with 10' wide hallways, the classes needed to be balanced. Fighters were simple - all weapons and armour were allowed. Priests were more limited in weapons, in exchange for spells at 2nd level. Wizards got spells at first level and were more limited in melee as a result.
In AD&D 2e, the wizard is limited due to the explanation that he doesn't have the training to wear armour and wield weapons (except for those that require little strength, little training or both). This explanation, while better than none, is less than perfect.
Later (when ASF was introduced), the idea of armour getting in the way of gestures came along as explanation. This is, in modern D&D, the accepted standard, along with ways for wizards to get around it (Mage Armour, special gear, etc.).