Before I begin... I apologize for this wall of text. This is a difficult spell, after all.
Comparing Level 1 Exhaustion with Hex
I'll start by citing a spell combo I used to do in an old campaign. I would cast either Heightened Maximilian's Earthen Grasp on a low Strength, high Int enemy; or Heightened Phantasmal Force on a low Int, high Strength enemy (usually the monsters aren't made with both stats high).
Now, both spells are notable because they are gated by a saving throw, but once you've failed, you need to make an ability check to succeed. As both MEG and PF are concentration spells, I could only hope the spell stuck long enough to be useful. The upside is, of course, the ability checks these spells require do not add proficiency.
Anyway, a party mate of mine would follow up with Hex on the target I've cast it on. This virtually prevents them from escaping the spell. On top of that, I had a save DC of 17 at level 5 (20 Cha and magic item). So even something with 20 in a stat, once they failed the first saving throw, would still find it utterly difficult to escape. And I was making sure I targeted their weaker stat for this as much as possible, stacking the dice entirely not in their favor.
The effect? We killed a Boulette with 200+ HP (it was something like 260 HP, but I'm not sure exactly) without anyone falling unconscious, and indeed, with only 1 member of the party getting attacked at all. The DM had been foreshadowing us that this was going to be a tough fight, too. And it would have been: it had 2 or 3 attacks a round, and each one did 50 damage on average! This was on top of resistances. However, the Boulette could not move due to MEG, so our ranged fighters and fighters wielding reach weapons were easily able to pick it off.
We also maimed a homebrew snake-like monster who was caught up in a PF. This one was a harder fight as it wasn't alone, but this particular creature was severely delayed (wasting ~5 rounds) and spending its action to do Investigation checks on the illusion before it broke free.
(That said, PF is much stronger than MEG, and the most utility I got from it was either outside combat or as a combat ender; as in, after I cast PF, that was also typically the end of combat)
Comparing Level 1 Exhaustion with Menacing Attack
In a different campaign, I quite enjoyed my Battle Master Fighter/Barbarian who would inflict the Frightened condition on his targets with Menacing Attack, and use a bonus action to grapple with Tavern Brawler. The Frightened condition imposes disadvantage on ability checks. This combination means I am always rolling Athletics with advantage, and the target rolls with disadvantage if they failed their saving throw. A further multiclass in Rogue gave Expertise in that skill. So, pretty much nobody was ever escaping my grip.
Many fond memories there. Grabbing pirates and throwing them overboard. Grabbing wyrmlings and holding them in place to keep them from flying away. Very memorable.
One thing I didn't get to do, though, but had been wanting to do, was team up with the party Druid and have her cast Spike Growth on me. And then I would have walked to just outside the border of that spell with a grappled opponent, but keeping them in the area of Spike Growth. With double Dash plus movement, I would have gotten 45ft in total for a round. By strict RAW, that would be 18d4 damage with no save or attack roll on the grappled creature.
Best of all, the visuals of it: prone-dragging someone's face in an area of Spike Growth feels powerful and awesome.
On Getting One Level of Exhaustion
The point I'm making from above is, disadvantage on ability checks is not something to be scoffed at. A player who takes this spell will be taking it for many reasons, and they will have many strategies up their pocket to ensure they get the most use out of that exhaustion level.
Aside from grappler builds, spellcasters will have many options like the aforementioned MEG and PF, Web, Telekinesis, Bigby's Hand, Evard's Black Tentacles, etc. These all force affected creatures to make ability checks against an effect you want to impose against them.
From my experience, targeting ability checks is a smart move. It is often not resisted, and many effects that affect attack rolls and saving throws do not affect ability checks. Even Restrained creatures, by RAW, can resist grapples with no disadvantage on the check. It follows that laying down potent effects that rely on these checks makes for some unconventional and juicy moments for the players.
Now, an effect that allows one level of exhaustion immediately gives all these strategies the green light. Unlike Hex, it requires no concentration, so a caster can lay on MEG or any of the other spells which depend on ability checks on top of that. And unlike Menacing Attack, it disallows a saving throw, so non-magical grappler builds do not even need a Battle Master maneuver and can freely multiclass into Barbarian without worrying about the concentration requirement conflicting with their Rage.
The above strategies are not bad. In fact, they can be very fun and create moments of awesome. But all this to remind you that players can be exceptionally creative, and if they take this spell, they will have a plan in place to maximize even their single cast use of it. Strategies that neither you nor I will have anticipated will undoubtedly spring from this.
Rather Accurate Secondhand Knowledge: Critical Role
Critical Role is a Twitch stream about nerdy voice actors playing Dungeons and Dragons. One thing about it is that it has run for more than 2 years now, and every session is recorded. This means, not only have the players experienced a whole slew of challenges from the DM, but the audience also sees and learns from their experiences.
For Critical Role fans: spoiler territory ahead.
In one story arc, the Frenzy path barbarian acquired a cursed sword named Craven Edge. The power of this sword was such that, on a hit, the target must make a saving throw. If it failed, its Strength score was deducted by 1 and the wielder's Strength was increased by 1. The benefits disappeared on a short or long rest.
The barbarian did not know the maximum Strength limit this sword afforded him, but he was able to use it to get to 20 Strength per battle (up from his natural 17). One day, he decided to find out.
Following a combat in which he used Frenzy (and thereby acquiring a level of exhaustion), he elected to not sleep or rest, to prevent the benefits of Craven Edge from disappearing. This earned him another level of exhaustion.
Because of his barbarian speed and Mobile feat, the halved speed was barely noticed. So, I can't provide commentary based on real games on how half speed impedes you (though it sucks to be slow).
The following day, he used Frenzied Rage once more. After the combat, he maxed out his Strength score at 25 and gained another point of exhaustion. He now had 3 total, giving him disadvantage on all his rolls.
The party then made the mistake of taking a short rest. Now the cursed item's effects come in. Due to maxing out the cursed weapon's effects in the last combat, the barbarian now has to make a Con save. He rolls a 20 and a 1. The curse? If you fail the check, you die. The barbarian died then and there.
The Lesson: Level 3 exhaustion means game over.
Now, don't get me wrong. The above scenario is probably incredibly specific. But it is an example of exhaustion killing someone without the need to get it to level 6.
When all your saves are at a disadvantage, then that is as if all spells were Heightened against you. Even Ancient Dragons or magic-resistant Liches and Rakshasas will have a hard time fending off your spells now. And with all your attacks at disadvantage, you are as good as dead in an encounter.
Now, about finding the level of the spell...
This spell is unique, which makes it difficult to place. However, we have learned several lessons from the above example:
The first bullet point is there to remove any notions that blanket disadvantage to ability checks can be disregarded. At the very least, this informs us that it is at least a 2nd level spell, as it's better than Hex.
But the second bullet point is where we should truly scale this effect. The level that we want is something that will ensure that 3 castings of this spell is where the trade off happens. That is, we want to ensure that after three castings of the spell, no matter what, we've just spent enough resources to have killed someone had we cast spells other than this one, too. This applies for whether it's one person casting thrice, or three people casting once each.
Note: when I say "no matter what" above, I mean this spell should be treated as if it will always kill someone after 3 castings. This is because it has no saving throw. Other spells which require saving throws or attack rolls, which will be used for comparison, may not always kill someone after three castings.
Cone of Cold (5th level) can kill someone if they experience it thrice in a row. Each casting deals 39 damage on average, or 117 damage over three castings, but also over an AoE. But this is gated by a save, so this may actually fall to 58 damage if all saves pass, and it may become not enough.
Banishing Smite (5th level) might kill someone if they are hit thrice, and this one is gated by an attack roll. Each smite deals an average of 27.5 damage, and three hits will deal 82.5 damage.
Chain Lightning (6th level) might kill someone also. Each casting deals 45 damage on average, but it is gated by a save. This puts it on par with Cone of Cold, dealing 135 damage with three castings, but can be reduced to 67 damage if all saves are passed.
Disintegrate (6th level) might kill someone again. Each casting deals 75 damage on average, so three will deal 225 damage, but it is an all-or-nothing effect. On a save, the target suffers no damage at all.
Finger of Death (7th level) might kill someone after 3 castings. Each cast deals 64.5 damage on average, and is halved on save. This is the strongest spell so far. Three casts of this will deal 253.5, or 126.25 even if all three castings were saved.
And now we found it. Casting three Fingers of Death will always be lethal, even if they make all their saves, just like this spell. I'm stopping here because a player may not even make it to 100 HP unless they're a Fighter or Barbarian. However, if we wanted to keep going because monsters usually have higher hitpoints than 100, the only other spell that goes higher than this is Meteor Swarm.
Meteor Swarm (9th level) will absolutely kill people. Each cast deals 140 damage on average, and 70 on a save. Three castings will deal 210 damage even if all three were saved.
And so the spell level is...
In its current form, assign it between a 7th to 9th level, depending on the typical HP of your monsters. This is not there for its first casting effects, but for its 3rd casting effects. When you equate level 3 exhaustion to death, you must balance around that point because it is the peak of the spell, and not on the other times you cast it.
Final Notes
Adding a save to this spell means using it thrice will no longer "guarantee" death. This puts it within the realm of Cone of Cold and Disintegrate, which can kill someone on three failed saves, but will not do so if they pass those saves. In which case, the level can be between 5th to 6th level, but capping out at 6th level.
Obviously, this answer depends on some level of guessing as to the exact spell's level. Balancing around Level 3 exhaustion is a conservative choice. Level 4 exhaustion is effectively dealing damage equal to 50% of the creature's hitpoints that bypasses resistance and immunity -- potentially the single most devastating damage depending on the HP of the creature, and far stronger than Meteor Swarm. I'd advise you to simply not allow the spell to inflict up to level 4 exhaustion.
But if you choose to balance around this instead, then all the spell ranges potentially slide back by one (placing the spell in its current form in the 6th to 8th level range, and the spell with a saving throw in the 4th to 5th level range). This would probably be a bad idea as it means you are not equating Level 3 exhaustion with death, when a BBEG who has been imposed with such a handicap is probably going to die by the hands of the PCs rather quickly.
And remember: blanket disadvantage to all ability checks is very, very potent (remember the Spike Growth + Grapple-drag combo worth 18d4 damage per round!).