An 'ability check using your spellcasting ability' means making an ability check using the ability (Str/Dex/Con/Int/Wis/Cha) modifier associated with your spellcasting.
As an example of how this works in practice, a cleric who knows dispel magic suspects that one of their allies is afflicted by dominate person, a 5th level spell. They cast dispel magic on their ally to try and remove it.
Since they're using a 3rd level dispel magic against a 5th level spell, they need to make a spellcasting ability (specifically, in the case of a cleric, Wisdom) check (d20 + Wisdom modifier) against DC 15 (DC 10 + 5 (spell level of dominate person)). The cleric makes their roll and happen to get a total of exactly 15, meeting (thus beating) the DC and successfully dispelling dominate person.
Note that unlike ability checks for specific skills, there are very few ways to add your proficiency bonus to 'general' ability checks, so the caster's roll for dispel magic rarely has any proficiency bonus added to it. Features that may allow you to do so are things like the bard's Jack of All Trades, or the Abjuration wizard's Improved Abjuration.
Also, for clarity's sake, the DC the dispel magic caster is attempting to overcome is just a flat number based on the level of the spell being dispelled. There's no opposed roll involved in the dispel attempt/the caster of the 'opposing' spell does not get to make their own roll to prevent the dispel magic from having an effect.