Note: The following assumes that the core issue is that the location of the mage is unknown to the party, rather than simply that attacking someone that is invisible causes disadvantage. Knowledge of the mage's location makes these strategies less desirable, favoring things AoE attacks instead.
Control the battlefield in ways that break the mage's sight lines and give options for detecting the mage that don't rely on seeing it.
First off, the other answers are great and are totally viable approaches (Theik's in particular would be my first choice, as it's a straightforward counter). This is simply an additional option.
The first and most important thing to do is to frustrate the mage's ability to damage you. Spells are powerful and effective, but only a limited number are typically available during a fight and they require line of sight to be effective.
If your characters can prepare or find and exploit areas of total cover (like terrain features, spells like Fog Cloud, and similar) you can really disrupt the mage's ability to target your party with spells. AoE spells can still sting, but if you have cover over a larger area (like hiding in a trench, or running around behind a wall) even those can miss. If your party is level 6, this enemy is unlikely to have a huge number of spell slots, so they can't afford to miss very often and remain effective.
For that matter, buying a few more potions of healing beyond your normal kit would give you more ability to be hit by those spells, and once the mage uses those spell slots they're gone. Once the spell slots are gone, it won't matter much if the mage is visible (or even present, unless there's more to them than the magic).
The next thing to consider is that you have more options for determining a creature's location than just looking at it. How, specifically, you locate it instead depends on specific resources you may have and what the terrain is like. If you can cover a stretch of the ground in sand, mud, gravel, broken glass, water, or something similar, then even an invisible creature will give signs of its passage. Those might be footprints, sounds, or something else. Even if the mage can try a Stealth roll, you will have more chances to locate it than you otherwise would.
These two can also be combined. If you can prepare your battleground, or contrive changes to it when you're there, well-placed obstacles blocking sight can be used to create areas where the mage would have to be to target your party well and then those areas can be set up to give away an invisible occupant you'll have done a lot to neutralize the advantage.
I don't know the pre-combat setup, but do you need to rush in head-on? With a Shadow Sorcerer, Thief, and Druid you've got a lot of options for sneaking and stealthy approaches. A cautious approach capped by the right action (which will depend on what your party can pull off reliably) could neutralize the mage immediately and before combat takes place at all.
I'll especially recommend this as it seems that you've had several run-ins with this group of enemies and seen the same results every time. It's worth rethinking the frontal assault. I can't speak for your DM, but when my players start to settle into a particular battle strategy I give them a few encounters that frustrate it. Just to keep things interesting and prevent my players from getting complacent.