-28 to all ability checks
I will give this a try and assume for the time being I can bring a nearly unlimited number of friends(or foes). I find it surprisingly hard to really push the modifier down, but this is my shot at it.
Lets start with the easy part, roll unlucky and you get a score of 3 in Dexterity, giving you a -4 for any ability checks you make.
Now some enemies will cast a number of spells at our target (we assume all possible saves fail):
Magic items: -2
Now we grab ourselves some magic items to even further curse ourselves (quite literally):
Enemies and "friends" abilities: -12
Have a College ot Lore Bard use their cutting words feature on you, giving you a penalty of one roll of their bardic inspiration dice. We assume the bard is at least level 15, so their die is a d12.
With that, we reach a total of -28 to all ability checks.