Characters may not know that they failed, but they may try multiple times anyway, even if they don't know the success/failure state
There are three questions we need to ask:
- Can the characters try multiple times?
- If they can't then the whole question is moot
- What about consequences of failure?
- Do they know they have failed?
Can the characters try multiple times?
Yes
The DMG has this rule in the Ability Checks section of running the game (emphasis mine).
Multiple Ability Checks
Sometimes a character fails an ability check and wants to try again. In some cases, a character is free to do so; the only real cost is the time it takes.
[...]
In other cases, failing an ability check makes it impossible to make the same check to do the same thing again.
[...]
This rule presents two sets of cases, ones where the character is free to do so, and others where the act of failing has made retrying impossible.
The fact that there is a rule suggests that characters do know that they didn't find anything. For example the in character motivation may not be "I did a bad job of investigating" but rather "I haven't found anything yet but I'm sure there is something here!"
For a real life analog, have you ever been looking for something in a room, not found it, but then gone back later only to find it almost immediately? This could be the same thing.
The examples given in this rule are guides. For example, it would be hard to provide an ingame justification for trying an insight roll again without a change in situation, as the insight skill represents your characters "read" of another characters actions and motivations. Similarly if the History roll was to see if they knew a piece of obtuse information, they can't try again to see if they "remember better".
On the other-hand, redoing a physical check like an investigation or active perception check is precisely the type of thing that this rule is designed for. If they aren't under time pressure (either self-imposed or environmentally imposed) and they want to persevere at the task then the rules allow them to do so.
What about consequences of failure?
The same DMG section also has optional rules on Degrees of Failure and Critical Success/Failure. If you are implementing these rules then their individual failures may have specific consequences that make further attempts impossible.
For reference these rules are:
Degrees of Failure
Sometimes a failed ability check has different consequences depending on the degree of failure. For example, a character who fails to disarm a trapped chest might accidentally spring the trap if the check fails by 5 or more, whereas a lesser failure means that the trap wasn’t triggered during the botched disarm attempt. Consider adding similar distinctions to other checks. Perhaps a failed Charisma (Persuasion) check means a queen won’t help, whereas a failure of 5 or more means she throws you in the dungeon for your impudence.
Critical Success or Failure
Rolling a 20 or a 1 on an ability check or a saving throw doesn’t normally have any special effect. However, you can choose to take such an exceptional roll into account when adjudicating the outcome. It’s up to you to determine how this manifests in the game. An easy approach is to increase the impact of the success or failure. For example, rolling a 1 on a failed attempt to pick a lock might break the thieves’ tools being used, and rolling a 20 on a successful Intelligence (Investigation) check might reveal an extra clue.
The botched disarm a trap check example would obviously make future attempts to disarm simple traps both unnecessary and impossible (since simple traps don't automatically reset once triggered).
Similarly, if a theif rolls a natural 1 on their check to pick a lock and break their Thieves' Tools, future picking attempts cannot be made until/unless they source a different pick set. Alternatively it may increase the DC of pick attempts using that set of Thieves' Tools as that set has a key component that is damaged.
Do they know they have failed?
Usually, when a character is making an ability check they are attempting something that has a success state at the very least (or at least the possibility of a success state). They will certainly know they didn't succeed. Whether or not they know that they failed is a slightly different question though. The answer to that question is "not necessarily"
I'll illustrate this with two examples from my game, one where the failure state is unknowable (or at least unmeasurable) and one where the failure state is knowable (or measureable).
Failure State is unknowable/unmeasurable
My group is currently dungeon crawling through the Dungeon of the Mad Mage. This mega dungeon has secret doors/rooms dotted throughout it. Not every room they go into has a secret door however. In order to not give away whether or not there is a secret door in a room I ask them for Perception/Investigation rolls any time they say they are looking for secret doors or secrets.
- If they roll poorly, then regardless of whether or not the room has a secret door I say "You don't find any secret doors".
- If they roll well (above the predefined DC for secret doors in this dungeon) then they either get "You're pretty sure if there were any secret doors in this room you would have found them, based on your experience with this dungeon to date" or "You found the secret door".
If there is a special secret door with a particularly high DC to be found, but they reached the "regular" DC, they still get one of those two answers. In that case, they don't know whether or not they failed with a low roll, but they also don't know if they succeeded with a high roll, they just know they didn't find anything. With a high roll however they have a pretty good idea that there probably isn't a secret door.
In order to discourage repeated rolling in every room, each check of this kind has a time penalty which works like this:
- Each check to search a room takes 15 minutes for "regularly sized rooms" or "regularly sized chunks" of larger rooms. That means a repeated check uses up time, time during which the monsters on the level they are on could attack, or they could be doing something else.
- If they want to try 2/3 times, they just roll the requisite number of times and I adjudicate the result based on each roll (depending on time pressure) and mark off the time
- If they say we are going to search this room until they find something or get bored, I ask them for an amount of time that they want to allocate to this task.
- I then ask them to roll a single d20 roll with their proficiency bonus for each character performing the task.
- I then compare the results of those rolls with a predefined table I generated which translates the modified roll into a "how long it took" for each DC level in the game.
- Based on that set of checks I then declare either "they didn't find anything in their alloted time", or "xyz found something/is pretty sure after x amount of time that there is nothing in the room".
Failure State is knowable
As a counter example, if the rogue is picking a lock and doesn't fail bad enough to break their lock picks, then they know that they didn't unlock the door. The failure state in this case is obvious. In this circumstance they can always try, and try again until they break their whole set of tools, or they open the lock. They may even have come across this type of lock before (Halaster does batch lock orders of course :P) and know roughly how hard the damn thing is to open.
Conclusion
Multiple ability checks at a single task are possible and explicitly allowed for in the rules. Whether or not a character gets to do them depends on whether or not the task has been made impossible for them to attempt again by virtue of their initial attempt.
Whether or not the character has an IC motivation to do try repeatedly is more fuzzy. Ultimately it's a question for the player and ostensibly the player can say their motivation is that they are particularly suspicious of their lack of success at the task and wish to keep trying. Or their character can think that they didn't do that well at the task and they want to do better.
If you as the DM feel that trying some checks over and over again is tedious/making the game unfun for you, then you need to talk to your players. Either come up with ways to shorten how much game time is spent on the task (I've presented how I handle that above), or come up with some other solution in conjunction with your players.