If a player uses Divine Smite on a creature, and unbeknownst to them, that creature is a fiend or undead, does the character learn this? If not, how does the DM ensure that the rules are fair, without the player learning due to the extra damage?
This is entirely up to the DM.
The rules don't say anywhere whether or not a player automatically knows about whether or not their extra damage triggers on their attacks. The Divine Smite ability in specific doesn't say that, for example, an extra burst of holy light strikes the target, or any other obviously visible display.
Thus, I would say that this is up to your DM and the style of the game that you are playing. If you're playing a game where easily knowing that some of your enemies are undead or fiends would ruin the fun, then you probably do the extra damage invisibly. If you're playing a typical dungeon crawl and it doesn't particularly matter what type of enemy you're fighting, then maybe you could know that the extra damage is being dealt.
It doesn't really matter that much either way. You can always use your Divine Sense to figure out for sure whether they're a fiend or undead, so knowing that you're doing the extra damage wouldn't usually be a big deal.
If the DM decides that this information should be secret, then they have a few options. IF they have a way of secretly rolling the damage die without the player knowing (say, though an automatic die-rolling app, or behind a screen), then they can roll the damage normally. If they can't practically roll the damage without the player knowing, then they can apply the average damage (in this case, 4.5) without telling the player that the extra damage has been added.