This is an underestimate of the difference in damage you would expect with larger creatures.
In the 1926 essay "On Being the Right Size" by J. B. S. Haldane, he recounts how a mouse that falls down a mineshaft would survive despite a mouse being so small the slightest injury would end it but a horse that fell the same distance wouldn't just die but "splash" (his words, it was the 1920s).
Because the larger a creature is the more mass is above the point of impact so at any given velocity the impact with the ground will impart greater crush forces even though a larger creature is more sturdily built its weight works against it. A larger creature will also be slowed less by air resistance.
So I'd expect damage to be so much greater that even small falls would be very dangerous to large-size or larger creatures. Maybe instead of increasing the damage per 10ft, instead narrow the increment.
A medium creature like a big dog should be able to safely drop 5ft but a 5ft drop for a horse is not safe, it stands a serious risk of injury. For a huge creature like an elephant even a 2 foot drop risks injury and 4ft is a much greater risk.
Without this you may have strange phenomena where a horse can safely drop 9ft but 10ft suddenly takes huge damage.
Another reason this helps is that it's easy to get large numbers of d6 dice, so if you do need to calculate a lot of fall damage you can pour a load of d6 onto the table.
[EDIT]: To be clear, it would not significantly add to the realism or verisimilitude to have such a small change in fall damage such as going from 1d6 to 1d8 or 1d4 damage per 10ft with Small or Large size. The fall damage is only a 28% difference on average when on impact a large creature hit with double the mass per surface area, or half for small creatures. Damage should double or halve respectively.
The main gameplay effect would be to annoy the players that a different set of dice than usual would be used, particularly a type that people are less likely to have multiples of. It's a lot easier to get a great number of d6 dice than any other type of dice.
A design ethos for 5e is to avoid table lookups as much as possible but just memorize iterations on the rules. Yes, this is broken with the exhaustion table, and it's considered a flaw in 5th Edition's design that's a typical topic of discussion of how to correct it.