Characters know when they fail an ability check
What is an ability check?
The DMG tells us:
An ability check is a test to see whether a character succeeds at a task that he or she has decided to attempt.
The PHB gives us the following guidance:
An ability check tests a character's or monster's innate talent and training in an effort to overcome a challenge. The DM calls for an ability check when a character or monster attempts an action (other than an attack) that has a chance of failure. When the outcome is uncertain, the dice determine the results.
As a result an ability check is a measure of a character attempting to succeed at a given task or challenge. In order for someone to attempt to succeed at a challenge or a task, you have to know the task is available to be attempted.
Does a character know they failed a check?
Yes.
The PHB gives us the following details on the mechanics of a check:
If the total equals or exceeds the DC, the ability check is a success — the creature overcomes the challenge at hand. Otherwise, it's a failure, which means the character or monster makes no progress toward the objective or makes progress combined with a setback determined by the DM.
In addition the DMG gives guidelines on trying a check multiple times (which requires the knowledge that you have failed):
Multiple Ability Checks
Sometimes a character fails an ability check and wants to try again. In some cases, a character is free to do so; the only real cost is the time it takes. [...] However, no amount of repeating the check allows a character to turn an impossible task into a successful one.
In other cases, failing an ability check makes it impossible to make the same check to do the same thing again. [...] But you might decide that the initial failure makes those checks more difficult to pull off.
Both of these passages, particularly the multiple ability checks passage, make it clear that characters know when they fail at something they have attempted.
But surely there are situations where a character doesn't know they have failed a check?
Of course there are. For these situations the PHB provides:
Passive Checks
A passive check is a special kind of ability check that doesn't involve any die rolls. [...] or can be used when the DM wants to secretly determine whether the characters succeed at something without rolling dice, such as noticing a hidden monster.
How does this apply to Dispel Magic?
The checks in Dispel Magic are clearly active checks as they involve a die roll. In active checks the character will know if they have succeeded or failed. As a result they know that they have succeeded or failed for each time they have to make this check.