Hex won't cause disadvantage on the saving throw
Ability checks and saving throws are different things.
Also, choose one ability. The target has disadvantage on ability checks with the chosen ability. (PHB, p. 251)
Not casting hex, since it isn't helping you, removes the worry about being invisible for that strike so that your hit can be a critical hit.
Surprise is ruled by the DM in any case (see the question here), but you have set it up so that you should surprise the target; you'll have advantage if you strike first. The key element here is setting it up so that "the target has not taken its turn yet" as a precursor to the strike. (Assassinate; 3rd level rogue assassin ability; PHB p. 97).
The DM will likely call for a Dexterity (Stealth) ability check as you set this up; success should set up surprise. While invisibility should make you unseen (unless the target has true sight, or a similar ability) you still need to be undetected by hearing, and possibly by smell. Both of those senses are mentioned in some monster stat blocks as providing advantage on Wisdom(Perception) checks.
Is the enemy surprised even though I used hex before my attack?
Solved, see above.
Is there anything wrong with this order of events?
Hex won't help you.
Is my calculation also correct?
You appear to have multiplied it by two, twice. Since serpent venom can be saved against, you don't double that.
If the target misses its Constitution save, your numbers are correct (except for the doubling of poison) from the death strike ability. The target may make the save, since the dice can be fickle.
If the target does not make their constitution save versus the 17th level ability:
(2d6 hex + 18d6 sneak attack + 6d8 fire + 5 dex modifier + 2d8 rapier)*2 + 3d6 poison (half for save).
But, see below, I think the case can be made that the poison damage goes inside the parens.
Aside: whether or not two saves have to be rolled, or one, since both are Con saves, is an interesting question. I'd recommend separate saves since two different mechanics are in operation here.
Do we double the poison delivered by Death Strike? I argue yes.
An argument can be made for folding the 3d6 (half for save) inside the parens since it is part of the whole attack; Death Strike isn't a critical hit, but is rather a unique rogue / assassin ability. That would resolve the damage as:
(2d6 hex + 18d6 sneak attack + 6d8 fire + 5 dex modifier + 2d8 rapier + 3d6(half for save)*2.
I believe that this is consistent with the rules text for Death Strike as compared to rules text for Critical Hit.
Poison is usually treated as separate from weapon damage dice on attacks, and crits, in terms of what dice are dice rolled for doubled damage. Since poison involves a saving throw effect, it doesn't look like doubling the dice fits in that order of operations for a critical hit.
Q: If you crit with a poison coated weapon, do you double poison dice because crit, or not because saving throw?
A: The intent is no. The saving throw, not the attack, determines whether the poison takes effect after a hit.
I'll offer that this ruling need not apply to Death Strike. Injury poison (in which class serpent venom falls), does some damage on any hit, dice are rolled for damage, and it does full damage on a failed save. (DMG p. 258)
The problem with ruling the poison outside the parens Death Strike is that Death Strike isn't explicitly a critical hit; it is a class feature contingent upon a save made by the target, like a "save or suck" effect along the lines of Disintegrate or Banishment. Doubling the damage of the poison (placing that 3d6 inside the parens) on the Death Strike is consistent with the text of that rule.
Since Death Strike is a different mechanic than critical hit, it does not have to follow critical hit's rules. (Crits don't allow for a save, crits don't fold in ability modifier, and per Crawford above, crits don't fold in poison damage ... ).
On a failed save, double the damage of your attack against the creature. (PHB, p. 97)
So double it all.
My reasoning is that the rogue's attack resolved by Assassinate, if surprise is achieved, ends up like this:
(2d6 hex + 18d6 sneak attack + 6d8 fire + 5 dex modifier + 2d8 rapier + 3d6(half for save).
On a failed save versus Death Strike, you double that, since it doesn't say you roll extra dice, but it says double the damage. It doesn't make exclusions the way that critical hit does. Critical hit says ...
When you score a critical hit, you get to re-roll extra damage dice
for the attacks damage against the target. ... and then add your relevant ability modifier(PHB p. 196)
(2d6 hex + 18d6 sneak attack + 6d8 fire + 5 dex modifier + 2d8 rapier + 3d6(half for save)*2 is "double the damage" literally.
Based on the ruling cited, I can see the argument against this being that "poison" is making its own attack to be saved against. What the assassin rogue is doing is providing a way for the poison to do it's thing; for serpent venom you need to do piercing or slashing damage to let it work. (DMG p. 258) While that's the general rule, and the ruling fits for most situations, I'll offer that Death Strike is a Specific beats General (PHB, p. 7) case of doubling all of the attack's damage, based on how the text is presented.