A spell caster wants to teleport to somewhere they know, that's outside their range, but they don't know they are too far away. Do they learn they're too far away if they try? Does the spell just fail? Do they end up at somewhere like it within range?
The spell fails and is wasted.
According to page 171 of the Player's Handbook, "If you ever try to cast a spell in conditions where the characteristics of the spell (range, area, or the like) cannot be made to conform, the casting fails and the spell is wasted."
The definition of "spell characteristics" given in that quote is the only one we have, but it seems reasonable to assume that the maximum distance the teleport spell can achieve is one such spell characteristic. It is, after all, a characteristic of the spell.
Teleport spell is a bit special. You said that caster knows the place but not the distance. And that distance to the intended place is too great. But distance the spell is supposed to cover is not always distance to the target caster meant! If we assume that his familiarity with the place is "seen casually", he rolls d100 with following results:
- 01-88 – on target – but spell can't do that, so nothing happens. I encourage the DM to narrate this nothing in an interesting way.
- 89-94 – off target – there is random distance roll. If the circle of offtargetness overlaps with distance spell can cover, there's no need for spell to fail. If it can teleport up to allowed distance, and be rightly off target, then I believe it should.
- 95-98 – similar area – if there is one within the allowed distance, there's no reason for the spell not to do this. Text of the spell seems to assume there always be one.
- 99-100 – mishap – this result requires no distance at all, so again, no reason for the spell to fail (yet). If the second roll would indicate on target, I don't know rules-wise what should happen, but personally I'd rule that caster took damage and bounced back to the point where he began. Interpretation that spell failed and damage never really occurred would be valid, too, but I'd rather give player damage, knowledge and interesting event than nothing.
To answer your question:
Do they learn they're too far away if they try?
Sadly no, nothing in the rules indicate they do. However, my experience in DMing tells me that it would be a boring result not moving the story forward. This tends to be disappointing and boring to players. Some of my earlier games stalled when 2 or 3 ideas failed without doing anything, players felt like "why should we try? If there's only one thing that will work we don't want a guessing game.". To avoid that, failures should do something, if ever so slightly.
Thus, I would probably try to narrate the failure in a way that would make them think if distance is the problem. Just not too directly. But I would break the rules as written by doing so.