You don't get to use a single spell to make a serious challenge for a party of 4 equal-power adventures lose the fight.
Monsters get high and increasingly strong saves and legendary resistances. They get bigger (nullifying a number of suck moves that work on smaller targets), and have more veto abilities (like teleports).
To beat them with spells that involve saving, you have to shoot volleys of "this would be bad if I didn't save against it" spells from every party member, and ideally from summons as well, with as high save DCs as you can pull off.
The goal is to deplete the monster's legendary resistances, or to simply land an annoying condition through sheer luck.
There are level 1 spells that are nasty enough that a monster wouldn't want to be subject to them -- faerie fire, bane, hideous laughter for examples.
A non-wizard can Conjure Woodland Being, then have 8 pixies all cast polymorph (turtle) on the enemy. If their wisdom save is less than +11 this has a pretty good chance of nullifying the foe's offence or eating a legendary resistance or two. An adult red dragon has a +7 wis save; they need a 5 to save. With 20% failure chance per pixie, that is 1.6 legendary resistances burned per cast of Conjure Woodland Being. (Being a turtle doesn't win the fight, but it is pretty close.)
As the party barrages the foe with things to eat legendary resistances, you'll get a better idea of what saves aren't insanely good. You'll want a collection of "save or suck" spells that target 4 to 6 different stats prepared.
Then you can start dropping those on it, focusing on its weakest stat. Usually the spell will fail, but a repeated 30% chance to win fight is a pretty strong move.
Another approach is to give up on save-or-suck spells. Defending and increasing the offense of your party is a tried and true way to win.
There are a handful of spells that don't rely on saving throws. Irresistible Dance burns an enemy action regardless of what they do (and possibly a legendary resistance). Force cage is a no-save no-concentration spell.
On a large or smaller creature, a warlock/sorcerer/wizard casting sickening radiance followed by a readied force cage is a "you lose" spell combo (please succeed at 95 out of 100 con saves at cleric spell save DC or die); if they are a spell caster, adding silence (requiring a 3rd spell caster) finishes the deal (a high level fighter-bard can do this themselves, which is gross: bard 18/fighter 2 does wish:simulacrum, double action surge, silence, sickening radiance, forcecage).