If there is no time pressure, rerolling is probably OK. The penalty for failure in this case is that you spend extra time doing it. The more important question is "Was the roll even necessary?"
This actually brings us back to one of the key points in the advice for DM in the game manual. It looks like a throw away line, but it's a good guide for when to call for dice rolls and when to let something simply happen.
In cases where the outcome of an action is uncertain, the Dungeons & Dragons game relies on rolls of a 20-sided die, a d20, to determine success or failure. (Basic D&D Players p4 emph mine).
So ask yourself when you come to a situation like this "what happens if they fail" if the answer is "they try again in a few minutes" then you probably just want to let the attempt succeed (provided there is no time pressure). If the answer is "they have to go another route" then you probably call for the roll and do not allow them to reroll it. If the answer is "the creatures in the next room are aware of them" then you allow it, but instead of calling for a reroll, failure is success with the complication that they made a bunch of noise.
Otherwise, it’s a failure, which means the character or monster makes no progress toward the objective or makes progress combined with a setback determined by the DM (Players Basic 58, emph mine)(HT to Okeefe for pointing this quote out)
So you have a number of options in the event of a failure that are not simply allowing your players to reroll the check:
- Failure leads to another route in the adventure.
- Failure means no treasure in this room (avoid this one, it's petty).
- Failure means they've lost important time.
- Failure means success with a complication.
The important thing here though is that you've run into a trap, you've asked for a roll where failure is meaningless. Don't do this.
When you ask for a roll, be prepared for both success and failure.