As a half-orc Fighter, I multi classed to rogue. At first rogue level with a rapier and shield, when applying the Savage Attack feature of a half orc, should do additional damage.
Normal damage
1d8 + mod and 1d6
Critical damage
3d8 + mod and 3d6
Is this correct?
What if my half-orc advances his rogue levels to 5?
Normal damage
1d8 + mod and 3d6
Critical damage
3d8 + mod and 9d6
Is this correct?
No, that's not correct. Half-Orcs get the following ability:
Savage Attacks. When you score a critical hit with a melee weapon attack, you can roll one of the weapon’s damage dice one additional time and add it to the extra damage of the critical hit.
As it says, when you get a crit, you get to roll one of the weapon's damage dice an extra time. So the first part of your calculation, the 3d8 damage your rapier will deal with a crit, is correct. However, Sneak Attack isn't part of the weapon's damage dice, it's extra damage that is added to the damage of the attack. Therefore, it's unaffected by Savage Attacks.
So your calculations should look like:
\begin{array}{c} \text{Level} & \text{Normal damage} & \text{Critical damage} \\ \hline 1 & 1d8 + mod + 1d6 & 3d8 + mod + 2d6 \\ 5 & 1d8 + mod + 3d6 & 3d8 + mod + 6d6 \\ \end{array}
