To answer the second part of your question, Treantmonk has made a video on YouTube wherein he shows his method for calculating expected damage that includes hit and crit chances, as well as the effects of (dis)advantage. While this is intended for DPR, it can work just as well for burst rounds.
Briefly put you take the average amount of damage you would do on a hit (Hex + Smite would be 34 according to my calculations), and then multiply it by your chance to hit (70% or 0.7 with a +8 to hit vs 15 AC). This example would yield 23,8 expected damage. Then you do the same for crits: take the additional damage a crit would offer (26 in this example) and multiply that by your crit chance (10% or 0.1) and add it together for a total of 26.4 expected damage on your first hit.
Using this method I have calculated two burst options for your character in an Excel spreadsheet, one where you use Hex and 1 Smite, another where you use both slots for Smite. Technically you could combine the two by maintaining Hex throughout a short rest to regain your spell slots, but I didn't include that.
The results are listed below in the following order:
The first number in each category assumes no attacks have advantage, the second assumes the first strike hits and knocks the enemy prone (most likely outcome), giving advantage to the following attacks, and the third number is for when all attacks have advantage.
Hex + Smite: 71,45(no advantage) 87,1875(advantage after 1st) 96,6675(full advantage)
Double Smite: 76,525(no advantage) 95,42625(advantage after 1st) 105,8813 (full advantage)
Now the Double Smite option is unlikely to ever come up as just one more round of using Hex would outperform it, but I thought I might as well include it as it is technically (slightly) more damage for that one round.
I've received a request to list the full formula, rather than relying on a video, so here we go. As I'm more comfortable with explaining things when I have an example, I'll use a 1st level character wielding a nonmagical rapier with 16 Dex and no fighting style attack AC 14.
Baseline:
First calculate your average damage on a hit. To do this you calculate each die as half its highest possible roll plus 0.5 (so d8 = 4.5) and add any additional modifiers such as ability scores or fighting styles. Our example would end up with 7.5 (1d8 = 4.5, +3 = 7.5). We'll call this number "Dmg". Then you convert your chance to hit to decimal (5%=0.05). Our example with his +5 on attack rolls attacking an AC 14 target would have to roll a 9 or higher on a d20, giving him 60% chance to hit, or 0.6. This number will be called "Hit". Now we multply Dmg by Hit: 7.5 x 0.6 = 4.5. We'll call the resulting number "Base".
Dmg x Hit = Base.
Criticals: Calculating the influence of possible crits works much like the calculation for base. We calculate the additional damage that would be added on a crit (1d8 = 4.5), which we'll call CDmg, and we multiply this by our chance to land a critical hit (1 in 20 = 5% = 0.05), called CHit. 4.5 x 0.05 = 0.225 and this number will be called Crit.
CDmg x CHit = Crit.
If we now add Base and Crit together (4.5 + 0.225 = 4.725) the resulting number will represent your average damage per round under normal circumstances.
(Dis)advantage: In order to calculate advantage we'll need two new values called "Miss" and "CMiss" which are equal to 1 - Hit and 1 - CHit respectively (1 - 0.6 = 0.4 and 1 - 0.05 = 0.95). These values should be directly linked to their associated hit counterparts, meaning that when Miss changes, Hit changes to match and vice versa. Now when we calculate advantage, we square Miss and CMiss (0.4 x 0.4 = 0.16 and 0.95 x 0.95 = 0.9025) and treat the resulting numbers as their new values, thereby also changing the values of Hit and CHit respectively (Hit would now be 0.84 and CHit is now 0.0975). In case of disadvantage you square the values of Hit and CHit (0.6 x 0.6 = 0.36 and 0.05 x 0.05 = 0.0025) and treat the resulting numbers the new values for Hit and CHit respectively.
Extra attacks: If you have multiple attacks in a round and all those attacks have the exact same modifiers and values, you can simply multiply Dmg by the amount of attacks you make for simplicity. If any modifiers change between attacks, for example wielding two different weapons, or one attack has advantage while the other does not, just calculate the attacks individually.
In slightly more formulaic form
Dmg x Hit = Base. CDmg x CHit = Crit. Base + Crit = Total.
With advantage: Miss x Miss = Miss. CMiss x CMiss = CMiss.
With disadvantage: Hit x Hit = Hit. CHit x CHit = CHit.
Example using this formula with advantage
0.4 (Miss) x 0.4 (Miss) = 0.16 (new Miss, changing Hit into 0.84). 0.95 (CMiss) x 0.95 (CMiss) = 0.9025 (new Cmiss, changing CHit into 0.0975).
7.5 (Dmg) x 0.84 (Hit) = 6.3 (Base). 4.5 (CDmg) x 0.0975 (CHit) = 0.43875 (CRit). 6.3 (Base) + 0.43875 (Crit) = 6.73875 (Total).
