I'd take two levels in Druid and two levels in Rogue
Two levels in Rogue gives you an ASI, Uncanny Dodge, and an increase in sneak damage. Two level in Druid gives you another ASI, a cantrip and access to 3rd level spells plus more spell slots.
So, Uncanny Dodge lets you halve the damage of an incoming attack using your reaction, which will help with survivability. Also the extra sneak damage is nice for increasing your damage output against single targets.
Your 3rd level druid spells gives you access to a variety of options, such as conjure animals (giving your foes 8 more targets really helps with survivability), and gaseous form from your Circle of Spores subclass, which is a good way to escape if things go badly.
Your ASIs could either bump DEX (increases attack, damage and AC), or take the Tough feat for a big chunk of extra HP (better taken later though, since at level 8 that's 16 more HP but at level 11 that's 22 more HP; doesn't matter that much though, since it'll end up giving you 22 more HP by 11th level either way).
Defensive Duelist also gives you a way to increase your AC by +4 (from 9th level onwards), but it uses your reaction so it's competing with Uncanny Dodge, so maybe not the best choice.
Finally, your 2 Eldritch Invocations; unless you have magical armour, then using mage armor via Armor of Shadows will be your best option for AC (if you increase DEX with an ASI, it will also out-compete medium armour like Half-Plate, and doesn't affect stealth or make your armour a target for heat metal).
There's also Fiendish Vigor, which lets you have 5 temp HP via false life, but it will take up your action, so maybe something you want to pre-cast rather than rely on in combat. That said, this won't stack with your temp HP from Symbiotic Entity from your Circle of Spores, so this might be a bit wasted on you.
A better choice might be Repelling Blast, which you can use to push enemies away with eldritch blast if you're trying to create distance between you and them but still deal damage.
Alternatively, replace the levels in Rogue for two level in Warlock
As an alternative, you could swap the two levels in Rogue from my above suggestion for Warlock, which means you lose Uncanny Dodge and the extra sneak die, but you gain a Pact Boon (you get an ASI either way).
Although I dismissed Pact Boons above, the Pact of the Chain combined with the Gift of the Ever-Living Ones invocation means that you can recieve max healing. This, combined with cure wounds/healing word from Druid means that you can just give yourself a flat 11 HP (from a 1st level cure wounds, for example), which is good for survivability.
You lose out on an extra sneak attack die, though, so this comes at the cost of increased damage output. Also, you lose Uncanny Dodge, as I said above, but arguably this now makes Defensive Duelist look more appealing.
