Three points to consider here.
I think that "Earth Mastery" has a lot of resemblance to mythic power of Antaeus. Similarities are more than passing so that could be inspiration for it. In all versions of this myth I stumbled upon, explanation of Antaeus's power - is his mother Earth aided her son.
We don't have mother Earth (Geya) as planet sized super-elemental in DnD ^ ^ but the core principle could stand: earth elemental is strong on earth because it gains support from some kind of external source... And huh - here comes Planescape (or just Planes) cosmology. Think about Negative Plane, Positive Plane - they had points of connection to Material world. You can find Planescape definition of Plane or Earth or use simpler version like Earth Plane. Place of stone pressure, metals. No wood or air. This could be source of Earth essence that fuels all earth elementals, 'a breath of earth' for them to speak metaphorically in environment that is mostly alien for them - the Material Plane.
What if their Earth Mastery is a matter of training with navigating through Earth Plane (see links above). In a sense that they are proficient with using everything you can find there - stone, metal, rocks... Not wood. How exactly? Maybe they manipulate themselves and opponent, maybe they hear vibrations... Because they are skilled and evolved to fight in that Earth hell - not a borrowed power, but matter of expertise and skill...
Earth elemental has an Earth Glide ability.
An earth elemental can glide through stone, dirt, or almost any other
sort of earth except metal as easily as a fish swims through water.
What if developers of DnD gave consistent names to abilities of elemental? What if "Earth" part means the same for Earth Glide and for Earth mastery - "stone, dirt or almost any other sort of earth except metal"?
Generally, considering all three points, I come to think that Earth elemental gets that bonus even if it's second floor of building, provided that floor is made of "stone, dirt or almost any other sort of earth except metal", and this floor is connected to the Earth Plane or greater Earth (which is connected to Earth Plane) by at least a tiny wire of earth (here - including the metal). If you treat source of power for Earth elemental as "2" (as predator with senses and training suited for certain environment) more than "1" (as mythic outsider creature gaining it's power from elsewhere) you could skip the 'wire to the Plane of Earth' part.