3.5e dominate person and calm emotions—no interaction
In 3.5e, dominate person is not a charm spell, but rather a compulsion spell—the two have different rules and different effects. For that matter, calm emotions has nothing to do with either unless they specifically altered emotions—so calm emotions could suppress the rage spell,1 also a compulsion, but does nothing against dominate person.
5e dominate person and calm emotions—total suppression
In 5e, dominate person applies the Charmed condition, and then also supplies an effect that reads “While the target is Charmed,” and calm emotions says it suppresses any effect that causes the target to be Charmed—which means that the “While the target is Charmed” clause is no longer met, so yes, 5e-calm emotions suppresses 5e-dominate person.
About being a “10th-level” spell—that sounds like a misreading
Dominate person isn’t a 10th-level spell in either edition—it’s 5th-level in both. However, in 3.5e, sorcerers get 5th-level spells at the 10th of that class—and thus the spell would have caster level 10th. That wouldn’t affect very much—just how close you have to be and how long it lasts. Meanwhile, 10th-level spells, as such, didn’t really even exist—there were rules for 10th-and-higher level spell slots, which meant you could cast lower-level spells from those slots for increased effect,2 but it wasn’t the same as there being printed “10th-level spells.” There were also Epic spells, which counted as 10th-level spells, but used entirely different rules. So I kind of doubt that the module has dominate person as a true 10th-level spell, because that isn’t really a thing in 3.5e. It’s not impossible, but seems unlikely.
Whatever “10th-level” really means, it doesn’t protect against calm emotions
Whether we’re talking about caster level 10th or spell level 10th, that doesn’t really matter to 3.5e-calm emotions. For example, it does suppress all fear effects—even if they’re cast with caster level 10th, or Heightened to a 10th-level spell slot. Neither of those would matter, because calm emotions does not indicate any limitation that prevents it from suppressing fear effects based on level.
Conclusion: if using 5e-dominate person and 5e-calm emotions, should have total suppression
In 3.5e, calm emotions would not be a valid response to dominate person at all, but in 5e, it is. And in both editions, the spell level doesn’t matter at all for this purpose—calm emotions does not indicate any difficulty suppressing higher-level effects in either edition.
And I recommend using the 5e versions of both spells here, because you are playing with the 5e rules. In 3.5e, there were other, common effects that could block dominate person,3 but those aren’t available in 5e. You instead used the 5e spell that is supposed to protect against Charm effects, calm emotions. That was the correct choice, and it should have worked.
Probably; 3.5e is not always clear about what is and what isn’t an emotional effect. Calm emotions calls out the morale bonuses of the rage spell, but doesn’t mention the other effects.
Using metamagic, which weren’t a sorcerer class feature but instead feats that made spells use higher-level spell slots in exchange for some increased effect. 3.5e also does not have default effects for casting spells from higher-level spell slots—even basic things like a higher saving throw DC, which was based on spell level, required the Heighten Spell metamagic feat.
For example, the core 1st-level cleric spell protection from evil could block all domination—the 5e protection from evil and good only blocks the Charmed effects of certain creatures. And note the wording—like calm emotions, it blocks the Charmed status. This is the way 5e handles blocking domination, instead of having charm and compulsion as separate things.