Surface is one of those terms that has no clear definition in regard to RAW and usually isn't stated explicitly. The extend of the area, whether it be a minute area on the back of a shell, the sunny place of the world (the ones the Dwarves refer to as surface) or the watery surface of a lake, depends on the context. A few examples:
PHB page 48, Unarmored Movement: [...] you gain the ability to move along vertical surfaces and across liquids [...].
Here, the authors clearly mean an area that is big enough for you, whether you be a dragon, giant, human, or pixie, to walk on.
PHB page 92, Hide In Plain Sight: [...] you can try by pressing yourself up to a solid surface, such as a tree or wall, that is at least as tall and wide as you are. [...]
Here the authors are more explicit and refer to your size to define the area but give examples of the nature of the surface.
PHB page 231, Demiplane: You create a shadowy door on a flat solid surface that you can see within range. The door is large enough to allow Medium creatures to pass through [...]
Here the dimensions of the area that is covered by the surface is given in the second sentence, at least medium sized.
In your case the matter could be easy if you were to go by what the description provides:
PHB page 245, 246, Glyph of Warding: [...] you inscribe the glyph [...] upon a surface (such as a table or a section of of floor or wall) or within an object that can be closed (such as a book, a scroll, or a treasure chest)[...].
Note: The caster can choose whether he wants to cast the spell either on a surface or on an object. It is their choice and not limited by the object itself. After all you could cast it on the first page of a book and state as trigger "opening the cover page".
The examples refer to some well-sized objects which you could use as a reference and simply rule that smaller surfaces cannot be inscribed. You could refer to the intricate nature of the pattern and the lack of ability to inscribe these patterns on the back of something as minute as a shell. Furthermore, there is a possibility characters (PC and NPC) can detect glyphs using the right tools.
Same spell description: The glyph is nearly invisible and requires a successful
Intelligence (Investigation) check against your spell save DC to be found.
This would state that the glyph needs to be big enough to be seen. If characters were to start arguing that their glyph is especially small and requires a tougher DC, you could simply disallow it. Alternatively you could try to go the way of a good DM and allow it. However, you should then require characters make the necessary skill checks. Require a proficiency, for example, in Runecrafting and a hard DC to make them small.
To sum up: While everything hints to the glyphs needing to be large enough to be seen with an Investigation check (due to them being nearly invisible by magical means), you could allow players to craft tiny glyphs with the right tools, time, and proficiency. After all it would be rather neat to have a character become famous for his skills in exquisite and tiny rune/glyph crafting.