Make the actual attack only be part of the whole 'job'.
The actual assassination should only be a small part of the entire adventure. Otherwise, like you said, it comes down to die rolls and is relatively boring. If someone is worth assassinating, then they probably have means and a certain degree of awareness. They would also most likely know that someone is hunting them, and would thus take measures to prevent being taken out from an open window. That is where you build in your challenges that can check off all your boxes.
The Settings
If the person you're after lives in a one bedroom house in the middle of nowhere, then the mission is easy and boring. Instead, put them somewhere more difficult to get to:
- A floating tower 100 feet in the air is great, because then your Rogue will need the assistance of the rest of his party to get up to it, and the Wizard can feel like a cool person.
- An underwater cave guarded by sentient porpoises that enjoy wrestling would make for a fun skill challenge that the fighter could help with
- A bog-standard stone keep (maybe overgrown with lecherous plants that will sound the alarm unless they're flattered?) will mean lots of small encounters need to be navigated before getting to the main event
The Target(s)
I'll be the first to admit that a level 11 rogue isn't really objectively all that powerful compared to other classes in 5e, but a level 11 character is supposed to represent one of the best specimens on the planet, and as a DM it's your job to make the players feel powerful (even if they really aren't). The way the book is written, a level 11 character shouldn't really be going after regular mooks. That would sort of be like sending navy SEALs against jaywalkers. Instead, your targets should be powerful in at least one way to justify their being a threat. They should be extremely strong or intelligent or charismatic. They should have resources and they should be able to anticipate threats and react accordingly. They should have assets they can call on for help.
The strengths that the targets have are what make the mission interesting and fun and challenging. Give your targets abilities like blindsight or blindsense to make sneaking up on them harder, give them elite guards with high perception, have them have decoys all around their compound, make them actually be shapeshifters who usurped the position of the person your players think they're targeting. Give them something that makes them special, which in turn lets your players feel special when they finally win.
The Challenge
You said you have a rogue, a conjuration wizard, and a gunslinger/fighter homebrew. It looks to me like you have a range-heavy party with skirmisher tactics and moderate single-target damage (depending on what your wizard has for spells). Given this, build your encounters based on that. If the fighter has a high STR, make situations that call for a high strength check. The rogue is most likely a DEX rogue so they'll need help if something heavy has to be lifted, which makes the fighter feel powerful. A conjuration wizard probably has Fog Cloud in their spellbook, so put the players in situations where they absolutely cannot cross an area without something blocking vision. Then your wizard can 'save the day' by letting the rogue and fighter get further into the compound.
The Complaints
From your description of your players, it looks like at least some of them aren't going to like this. The rogue seems like they would be 100% happy to just stealth in, do the deed, and leave for the entire game. You need to find a way to do this while not making everything trivial. For instance, maybe you could have the main target have three vassals that need to be dealt with, and one or two of them can be taken care of 100% stealthy by the rogue. Then the rogue is a hero, but doesn't shut down your entire idea.
You can also look at the rules for stealth in the book for guidance. You keep mentioning that the rogue can use reliable talent to always beat stealth checks, but I would recommend you to read the stealth section again. Stealth is relatively weak in 5e, it's difficult to get, even harder to maintain, and even the world's greatest rogue will never really be all that great at it. A level-11 rogue certainly can't expect to pass every stealth check they come up against, and in fact if you put 4 or 5 guards in the room looking for them, the rogue is almost guaranteed to fail against at least one of the active checks. The archetype of 'sneak anywhere and never be detected' isn't really mechanically possible in 5e using the book mechanics. Not only is stealth difficult to maintain even in normal situations, there are a variety of ways to defeat stealth 100% of the time with spells (true seeing, fairy fire, etc), monster abilities (blindsight, blindsense, tremorsense, true-seeing, etc), and even basic mechanics (it's not possible in 5e to stealth across a well-lit room without the invisibility spell). A few torches makes it mechanically impossible to stealth past guards in most circumstances (especially if your smart target sleeps in a room with one heavy, well-lit entrance and a few guards).
I think optimally, you should look at your player's abilities and come up with situations that only one of them can solve. Then put those together to form your mission, and that way everyone needs everyone else, and the payoff is rewarding instead of boring.