There are no strict rules for precise time calculation
Unfortunately, "two PC teams racing" simulations doesn't work well in 5e. There are rules on precise tracking time for particular situations (like combat rounds or travelling pace), but in most situations a DM is supposed to measure time roughly. That will be subjective, hence, might not feel "fair".
Time is just another kind of resource, available to PCs
The point of any time restrictions in 5e adventures is not to make players racing each other. Variety of resources can give players opportunities to make meaningful decisions. For instance, a wizard can cast Leomund's Tiny Hut as ritual or he can save ten minutes but spend a spell slot. A rogue can search for traps thoroughly, but spend a couple more hours, or she can take the risks, probably lose some HP triggering the trap. Players can spend more time sleeping, searching for food and water, or they get one more level of exhaustion, etc.
Sometimes the rules give you precise timings (like +10 minutes when you perform a ritual), but in most cases the timings are up to the DM. The rules generally don't specify precise timings for task resolving, since a single skill check might mean very different things (a momentary decision was made, or the whole day was spent searching).
What to do?
So how can you resolve the situation? I suggest you not to focus on the precise time management, but make the game more fun to play. D&D rules is not a reality simulator. You won't achieve "fairness" by following the rules as literally as you can. To make the game feel "fair", let your players have agency. Let them make meaningful decisions. Let them face the consequences. When time matters - give them an opportunity to save more time by sacrificing other resources, or by making clever moves. Let them be creative.
Visualize the game time - draw a clock, or use counters - set a deadline, and give your players opportunities to save (or spend!) more time. Every saved hour must have a cost. An example:
— We decided not to sleep and spend 8 more hours for searching for the flower.
— That will probably be all-consuming. I'll ask anyone for the Constitution check (DC 15) and give one level of Exhaustion on fail. Do you do that?
— I'm searching for a large beast footprints.
— Okay, make a Survival check, with disadvantage, because it's already too dark to see clearly. You can wait for tomorrow though.
— No, I don't want to wait. I can use a torch, or cast Light, can't I?
— You can indeed, but it probably lures more stirges to this place. Do you do that?