If the attempt to disarm fails, combat continues as if nothing happened
A big chunk of confusion around this is because the creature that is attempting to disarm another is making an "attack roll". Therefore it is natural to assume that it is just another form of attack. But there are two other examples to show that this is not the case: shove or grapple.
Shoving a Creature: Using the Attack action, you can make a special melee attack to shove a creature... Instead of making an attack roll, you make a Strength (Athletics) check contested by the target’s Strength (Athletics) or Dexterity (Acrobatics) check (the target chooses the ability to use).
Grappling: When you want to grab a creature or wrestle with it, you can use the Attack action to make a special melee attack...Using at least one free hand, you try to seize the target by making a grapple check instead of an attack roll: a Strength (Athletics) check contested by the target’s Strength (Athletics) or Dexterity (Acrobatics) check (the target chooses the ability to use).
These clearly point out that the creature does not make an attack roll, but instead make a check. And that check is contested.
In this case, the disarm rule states:
A creature can use a weapon attack to knock a weapon or another item from a target’s grasp. The attacker makes an attack roll contested by the target’s Strength (Athletics) check or Dexterity (Acrobatics) check. If the attacker wins the contest, the attack causes no damage or other ill effect, but the defender drops the item.
Since this is a contest, we look up the rules of contests:
Both participants in a contest make ability checks appropriate to their efforts. They apply all appropriate bonuses and penalties, but instead of comparing the total to a DC, they compare the totals of their two checks. The participant with the higher check total wins the contest. That character or monster either succeeds at the action or prevents the other one from succeeding.
If the contest results in a tie, the situation remains the same as it was before the contest. Thus, one contestant might win the contest by default. If two characters tie in a contest to snatch a ring off the floor, neither character grabs it. In a contest between a monster trying to open a door and an adventurer trying to keep the door closed, a tie means that the door remains shut.
So the two creatures contest to see if one will drop a weapon. Roll dice, and whomever has the higher total wins. So in this case, the creature holding the weapon wins and can keep their weapon. But what about the attack itself?
Why are we using a weapon attack roll?
Contests are normally done via skills, so why is this saying attack roll? Easy, because there are too many options for performing the maneuver and attacking is a consistent value!
Consider, the attacker, Kermit, is trying to disarm Fozzy. How would they do it?
- If Kermit is a fighter type with a big mallet, they would just knock the item away with pure strength
- But if Kermit had watched Errol Flynn, he might try using dexterity and a rapier to spin the item away
- Maybe Kermit is a monk and sneaks past Fozzy's defense and uses wisdom to grab a pressure point making Fozzy drop the item
- Then again, Kermit might be a Hexblade and doesn't have a lot of strength or dexterity, but through charismatic will, knows how to just walk up confidently and knock away the item
- Kermit seems more like a Battle Smith Alchemist, who would use his intelligence to realize that based on the on the fulcrum point, if he leveraged the item in a certain way, it could easily be torn free from Fozzy's grasp
Do you see where I'm going? Every one of those methods could potentially work, but each one of them calls for a different skill check: Athletics, Slight of hand maybe, medicine, I'm thinking intimidation, and maybe just a straight up intelligence check.
So instead of trying to come up with a new check every time a disarm is performed, and describing how it would be done, just let the character use a skill that they should be good at, attacking. And that is what each of these characters are doing; attacking an opponent with the goal of disarming, not necessarily harming.
And since I've mentioned it twice in the comments, this is not an "attack" as you are not using your attack roll against the opponent's AC; you are in a contest against another creature's Strength(Athletics) roll. The attacker is not trying to break through plate armor, a shield, mage armor, a shield of faith spell, or any other AC modification feature. None of that comes into play. This is the same mechanic as Shove and Grapple, but using a creature's attack roll instead of a skill roll.
From a certain point of view
There needs to be a balance on disarming; you can attempt to do damage or you can attempt to disarm.
If you look at this from the other point of view. Every time an attack lands a blow, you would need to have the recipient perform a strength check to keep their weapon, because why not? If Kermit makes an attack, and even though Fozzy held on, he still takes damage, why wouldn't every attacker say, "I'm trying to disarm."?
Also, since this does not care about AC, it wouldn't matter how fortified the opponent is. That BBEG with an AC of 30? No problem, attempt to disarm and even if they hold on to their weapon, you'll somehow still do damage.
Or Fozzy could say, "I drop the weapon." Since the disarm succeeded, he doesn't take any damage. Then as a free action, either pick up their weapon or grab a new weapon.