The rules don’t specifically tell a DM when a check can be retried or not. In the Ability Checks rules (Chapter 7 of the Basic Rules or Player’s Handbook), when describing the results of a check, the rules say:
If the total [of the roll] equals or exceeds the DC, the ability check is a success--the creature overcomes the challenge at hand. Otherwise, it's a failure, which means the character or monster makes no progress toward the objective or makes progress combined with a setback determined by the DM.
Chapter 8 of the Dungeon Master’s Guide goes into more detail under “Multiple Ability Checks”:
Sometimes a character fails an ability check and wants to try again. In some cases a character is free to do so; [...] In other cases, failing an ability check makes it impossible to make the same check to do the same thing again.
So the DM is free to rule that the character has merely failed to stabilise their companion, and can try again, or that they have done something wrong, making it impossible for them to do so, at least for the length of the combat.
Personally, though, I think this is a harsh call. Simply failing the roll is no reason to stop a character from continuing to attempt to stabilise someone; the consequences for failure are possibly permanent, and I would never want to discourage a player from trying to save a friend, especially if the party doesn’t have access to magical healing and they’re willing to give up their action each turn to try again.
From a narrative perspective, stabilising someone is effectively applying first aid, and you should be able to continue trying to stop the bleeding or restart their heart or whatever else is involved. Perhaps ask your DM outside of play - before your next session perhaps - why they’re making this call.
It’s also worth noting that Medicine is a Wisdom-based skill, not an Intelligence-based one, which suggests failure is more about you not finding and addressing the worst wounds, rather than that you don’t know what to do - unless there’s something special about these wounds? The section on "Intelligence check vs Wisdom check", also in Chapter 8 of the DMG, says:
Wisdom checks allow characters to perceive what is around them (the wall is clean here), while Intelligence checks answer why things are that way (there’s probably a secret door).