It All Started With Gold Dragons
The first dragons able to change shape into human form were Gold Dragons. None of the others could. On pages 11-13 of Monsters and Treasures (OD&D, 1974, TSR) there were a total of six kinds of dragons: red, blue, white, black, green, and gold. The only metallic dragons in the game were described as "being a class unto themselves." They were the only Lawful dragons1 (the others all being Chaotic in the old law/chaos alignment dichotomy), could take on human form and could employ spells up to sixth level which was the highest spell level in the game at the time.
Spell ability for other dragons were limited: White, none; Black, 1st lvl; Green, 1&2 lvl; Blue, 1&2 lvl, and Red 1 - 3 lvl. However, spell usage was only available if the dragons rolled on the "able to talk" chance. Even an ancient red dragon could not cast the 4th level polymorph spell.
In Greyhawk (OD&D Supp 1), more metallic dragons were added.
The Platinum and Silver dragons could assume human form. Copper, Bronze, and Brass dragons could not. The basic dragons remained bound to wyrm form. (Which is fine, if you are a dragon).
Gary Gygax stated that there was but one King of the Lawful Dragons and but one Queen of Chaotic dragons and that "Women's Lib may make whatever they wish from the foregoing." (p. 35, Greyhawk). The Dragons had yet to be categorized as "Chromatic" versus "Metallic." The Queen's "color" (the king was Platinum) was "chromatic" since she had multiple heads: one each of the basic dragons (non-metallic).
The design intent is blatant in terms of who was superior among dragonkind: the metallic dragons.
1e AD&D carried this design theme over. Only Platinum, the Silver and Gold Dragons could appear in human form ... unless they were a spell using dragon. Depending upon the dragon kind this capability was variable, and only dragons who could talk had a chance to use spells. The metallic dragons had higher chances to talk, and to use magic, based on a percentile roll than the basic/chromatic dragons in general. To determine if a dragon could use spells ...
- If you are a dragon, can you talk? (Consult MM, see percentile roll)
- Second, can you use spells? (Consult MM, percentile Roll)
- Depending on point 2, could you use a powerful enough spell to assume human form? (Lvl 4 / polymorph spell is the entry condition)
- On the other hand: how often would you want to do that? You're a Dragon, for goodness sakes!
At that point (1e) older Red dragons could use 4th level spells and thus via the polymorph spell could take on human form, as could Tiamat who could use 5th level spells. Bahamut (Platinum/King) could use 7th level spells at that point in the game's development and also had the ability to change into human shape, like silver and gold dragons. (MM, 1e, p. 33).
BECM books retained this convention, and AD&D 2e likewise.
In later editions and supplements dragons, like cloud dragons and shadow dragons, didn't have human form either, but steel dragons did. The number of articles on dragons in Dragon Magazine is non-trivial, so I won't try to summarize all of them. While I don't recall ever reading a "why" on the difference in innate ability, it may have been in one of the articles(it's been a lot of years).
Why? Part 1 = Tradition
The most plausible explanation as to why this theme carried over into 3, 3.5, 4, and 5 is "inertia" or "tradition" in keeping the game design faithful to its origin. At least the later edition dev teams were consistent, when you go back to the origins of the game, in confining the shape changing "class ability" to metallic dragons because that is how they were originally folded into the game.
Why would the devs change that? The distinction is as old as the game itself for the namesake monster of the entire franchise.
I offer a small frame challenge here, and suggest that the question may be backwards, and that the dev question is why did they make it so that Golden dragons could shape change, not why the other dragons couldn't. The follow on is "why could lesser metallic dragons change into human form?"
(That still leaves open "why would later devs change that feature?")
Why? Part 2 = Borrowing from an eclectic mix of myths and legends
Dragon varieties are rooted in diverse Mythologies and Legends from different cultures
Part of the answer is that the golden dragon came from a different cultural / mythological base, China, just as Bahamut and Tiamat have a relationship with the mythology of Sumer/Babylon, and the Eurocentric dragons like the ones Saint George slew, the one Beowulf slew, Smaug, etc. (The illustration in the 1e and 2e MM's are both evocative of an Oriental/Chinese dragon).
In the mythological tradition that they were borrowed from by D&D, the gold dragons took on human shape, whereas in the mythological traditions that the basic dragons were borrowed from by D&D, they did not.
1 From the ENworld archived thread of Q&A with G.G.
Draco Argentum: Gary where did the idea for having the cromatic and metallic dragons come from?
Col_Pladoh: ... If you read the intro to the SLAYER'S GUIDE TO DRAGONS from
Mongoose, you'll see therein how I came up with the chromatic dragons.
Some colors other than red were needed, hues that would be harmonious
with their breath weapons. When I'd done the four new additions to the
CHAINMAIL red dragon, it seemed a good plan to have a LG dragon, one
of gold color that was based on the Oriental model. From that I
worked on the rest of the metallic species. Of course, such potent
creatures needed great leaders or masters, so I delved into mythology
for the names--Tiamat and Bahamut--and thus came up with the Evil
Queen and good King of the chromatic and the metallic dragons.