A Stealth check is appropriate for this, but probably Charisma (Stealth) as opposed to the usual Dexterity (Stealth). It should probably be opposed by a Perception or Insight check on the pursuers part.
Stealth is described as (emphasis mine):
Make a Dexterity (Stealth) check when you attempt to conceal yourself from enemies, slink past guards, slip away without being noticed, or sneak up on someone without being seen or heard.
You might, in this case, change the base ability required for the stealth check to something more appropriate, like a Charisma (Stealth) check. The DMG specifically allows for this in the section on Using Ability Scores:
Under certain circumstances, you can decide a character’s proficiency in a skill can be applied to a different ability check. For example, you might decide that a character forced to swim from an island to the mainland must succeed on a Constitution check (as opposed to a Strength check) because of the distance involved. The character is proficient in the Athletics skill, which covers swimming, so you allow the character’s proficiency bonus to apply to this ability check. In effect, you’re asking for a Constitution (Athletics) check, instead of a Strength (Athletics) check.
Often, players ask whether they can apply a skill proficiency to an ability check. If a player can provide a good justification for why a character’s training and aptitude in a skill should apply to the check, go ahead and allow it, rewarding the player’s creative thinking.
Why Charisma? Well Charisma is a measure of your personality. It states:
Charisma measures your ability to interact effectively with others. It includes such factors as confidence and eloquence, and it can represent a charming or commanding personality.
From what you have described, your characters are in a social situation, and one in which they need to interact with people (blending in to the point of non-notability is still interacting with the room).
This would be opposed by one of:
- Insight
Your Wisdom (Insight) check decides whether you can determine the true intentions of a creature, such as when searching out a lie or predicting someone’s next move. Doing so involves gleaning clues from body language, speech habits, and changes in mannerisms.
- Perception [either Wisdom (Perception) or Charisma (Perception)]
Your Wisdom (Perception) check lets you spot, hear, or otherwise detect the presence of something. It measures your general awareness of your surroundings and the keenness of your senses.
- Investigation:
When you look around for clues and make deductions based on those clues, you make an Intelligence (Investigation) check.
- An Intelligence or Charisma (Survival) check
- Intelligence for making deductions, or Charisma for how well someone could read the room.
- Survival for following someone through a crowd (akin to tracking someone using their footprints, you track them by other people's reactions to them)
- Wisdom (Survival) may even still be appropriate here.
So what caused the weird disconnect?
The bit that caused the problem, and was likely inappropriate is the Pass without a Trace spell. It's designed specifically for dark situations where silence is key.
Pass without a Trace states(emphasis and emphasis mine):
A veil of shadows and silence radiates from you, masking you and your companions from detection. For the duration, each creature you choose within 30 feet of you (including you) has a +10 bonus to Dexterity (Stealth) checks and can't be tracked except by magical means. A creature that receives this bonus leaves behind no tracks or other traces of its passage.
Pass without a Trace, in an urban setting, coincidentally makes you cause a significant disturbance in a crowd (as opposed to just regularly walking through a crowd). Imagine you were a commoner and a shadowy, indiscernible, mass of figures surrounds you making no sound (as the characters walk around you). What are you going to do...probably scream your head off at the terrifying mass that is around you!
5e does not have fluff text, it is rules text all the way down. Pass without a Trace doesn't make you invisible. It doesn't make the people on the street beside you not see you. It makes you not disturb your surroundings, easier to hide in the shadows and not make any noise.
By making it a Charisma (Stealth) check you still allow the characters to use their Stealth skill, but Pass without a Trace no longer gives a bonus to the check as it's not Dexterity (Stealth).