Our party disguised ourselves as merchants, but when the enemy appeared and we attacked them, the DM just said, "Roll for initiative."
Is there a surprise round in D&D 5e like there was in previous editions like 3e?
There is still certainly "surprise", though not quite a "surprise round".
See the surprise rules.
It is generally up to the DM whether there is a chance that one side (or some of the combatants) may be surprised.
The rules given are keyed towards one side being stealthy (rather than disguised) and so it becomes a Dex(Stealth) check vs a Wis(Perception) check. Initiatives are still rolled for everyone, but anyone surprised is unable to take actions on their first turn of combat.
For your disguise tactic, a DM might shake things up and ask for the party members to make Cha(Deception) checks vs a Wis(Perception or Insight) check of the enemy in order to determine surprise instead. But this would be up to the DM and veers away slightly from the surprise rules exactly as written.
There is no surprise round in 5th edition. Instead, everyone who did not recognize that there was a threat is "Surprised" until the end of their first turn, and they are unable to take any actions at all until they have taken their first turn. Additionally, the creature is unable to take any reactions (such as opportunity attacks) until the end of their first turn.
This means it's possible for an enemy who rolls well on initiative to go first, doing absolutely nothing other than ending their "Surprised" condition. This enemy can now take reactions as normal.
As G. Moylan mentioned in a comment, surprise is determined by the GM and not the players. While players certainly can influence the chances to cause surprise (or to avoid being surprised), it's still up to the GM whether that actually happens or not.
Surprise round doesn't exist in D&D 5e. It's called a Surprised Condition. According to the official rules:
Any character or monster that doesn't notice a threat is surprised at the start of the encounter.
That states that it is a condition rather than a "round".
The GM compares the Dexterity (Stealth) checks of anyone hiding with the passive Wisdom (Perception) score of each creature on the opposing side.
That means that if you want to surprise a target you must attack them stealthily and the DM will have to check if all targets fail the Passive Wisdom.