When designing many monsters for D&D Gary Gygax, Dave Arneson and others frequently ripped off source material from folklore and myths - and radically changed them in many ways (1). So there is no explicit source because it was altered from the source or sources. Unless you can find a quote from Gary, Dave or the designer in question, then likely that info is lost to the sands of time.
Duergar and Derro were both designed to be "dwarves that used magic". You will note in 1st/2nd edition and earlier, normal dwarves never used magic at all. 3rd edition and future editions would later allow dwarves to become wizards, but those of us who are "dwarf purists" argue that the only way a dwarf can become a wizard is if they have derro blood (see further below).
Duergar and Derro were also more intelligent than normal dwarves. Derro had Int of 13 to 18 in the 2nd edition Monster Manual, Duergar 8 to 18. Normal dwarves and mountain dwarves only had 11 to 12.
In the case of Derro, they can become magic using savants. (And if you read the "Flint the King" book from Dragonlance, half-derro and dwarves with derro ancestors can also become magic users.)
In the case of Duergar, their growth ability is magical. The following is a quote from the 2nd edition Monster Manual (2): "All duergar possess innate magical abilities of enlargement and invisibility. They can use these spells as wizards of a level equal to their hit points. Duergar can use enlargement to either grow or shrink themselves-"
And you are correct, growing larger would seem to be a hindrance in cave spelunking... but shrinking would be a benefit. The ability to shrink likely developed first, but since the spell is reversible they developed both magical abilities.
The fact duergar also possess invisibility makes them way more complicated. A race of size-shifting and invisible dwarves gives them a range of options for ambushes, stealth, combat and so forth.
1 As mentioned above they radically changed things from their source material. A prime example of this is the dtrow - an Old English / Nordic / Scottish monster, which was essentially evil dark elves who lived in caves and under bridges. The dtrow would later become the source of both trolls and drow elves (note the proper pronunciation of drow sounds like know instead of like now, which means many gamers now pronounce it incorrectly - partially because WotC decided to go with the now style pronunciation so they could copyright the name).
2 The 2nd edition Monster Manual and related Monster Compendiums are basically the best/most complete version of all the early editions and the source from which all that came after it is based. When researching anything about a monster, I would argue the 2nd ed MM is the best source to be looking at because later editions would often gloss over the details. I still use the older editions as an influence even when writing adventures for 5th edition, mostly because the 5E MM may get the stats right, but it goes into very little detail about the ecology, culture, background, mating habits, etc of monsters. And those details are important in my opinion when it comes to storytelling.
