First the saving throw, then damage reduction, then resistance, and finally the threshold
This answer will largely rely on two previous question and the rules on resistance/vulnerability:
These give us most of the rules we need to adjudicate the damage taken by your monster; in particular we know from the quoted rule above that resistance is applied last and, to quote the first question linked:
[...] You take off the damage reduction and then halve the damage [...]
We also know from the second question I linked that (direct quote):
You take 1/4 the total damage. You are looking at two separate events. The first event is whether or not you are hit with the 'full force' of the attack (your saving throw determines this). You can think of it like diving out of the way of a fireball, or ducking under a column of ice.
If you pass the saving throw, you take half (you successfully avoided some of the damage). If you don't, you take the full amount.
Now, after this, you resolve your resistances.
This lets us know that all of these can apply to a single instance of damage. Furthermore, notice that the saving throw will always be the first thing that happens; if we succeed on it, the damage will immediately be halved; if we fail it, the damage will not be reduced. Then we apply any damage reduction (such as the Heavy Armor Master feat), and finally we apply any resistances we may have.
The question left is when does the damage threshold come into play? Let's look at the rules on "Damage Thresholds":
[...] An object with a damage threshold has immunity to all damage unless it takes an amount of damage from a single attack or effect equal to or greater than its damage threshold, in which case it takes damage as normal. Any damage that fails to meet or exceed the object's damage threshold is considered superficial and doesn't reduce the object's hit points.
Thus this is simply a final check we have to make after having totalled up the expected damage. If that total meets or exceeds the threshold then the creature takes that much damage, if the total is less than the threshold then the creature takes 0 damage.
And so we've arrived at the following ordering:
- Check to see if the saving throw is passed or failed to calculate the initial damage
- Apply any damage reductions
- Apply any damage resistances
- Check to see if the damage threshold has been met/exceeded to calculate the final damage
Here is an example using your numbers assuming the creature takes 40 damage:
- They pass the save, reducing the damage to 40/2 or 20.
- They reduce the damage by 3 down to 17.
- They have resistance, reducing the damage to 17/2 which rounds down to 8.
- Their damage threshold is 7, which has been met/exceeded.
- They take 8 total damage.
Assuming the creature takes 33 damage:
- They pass the save, reducing the damage to 33/2 which rounds down to 16.
- They reduce the damage by 3 down to 13.
- They have resistance, reducing the damage to 13/2 which rounds down to 6.
- Their damage threshold is 7, which has not been met/exceeded.
- They take 0 total damage.
If you're interested in a mathematical formula/interpretation you may use the following where X is the initial damage:
$$ \left \lfloor{ \frac{ \left \lfloor{ \frac{ x }{ 2 } }\right \rfloor - 3 }{ 2 } } \right \rfloor $$
Another way to explain this formula is the following:
If the initial damage is 33 or less, the creature takes 0 damage.
If the initial damage is 34-37 (including the ends), the creature takes 7 damage.
If the initial damage is 38-41 (including the ends), the creature takes 8 damage.
If the initial damage is 42-45, the creature takes 9 damage.
And so forth...
Looking at the above facts we can see that another way to calculate this is to take the total initial damage, subtract six, divide it by 4, and round down. Or in a formulaic form:
$$ \left \lfloor{ \frac{ x - 6 }{ 4 } }\right \rfloor $$
Another way to arrive at this formula is to use the following equality to our previous formula, this equality holds true for all positive integers h:
$$ \left \lfloor{ \frac{ \left \lfloor{ x }\right \rfloor + k }{ h } } \right \rfloor = \left \lfloor{ \frac{ x + k }{ h } } \right \rfloor \rightarrow \left \lfloor{ \frac{ \left \lfloor{ \frac{ x }{ 2 } }\right \rfloor - 3 }{ 2 } } \right \rfloor = \left \lfloor \frac{ \frac{ x }{ 2 } - 3 }{2} \right \rfloor = \left \lfloor{ \frac{ x - 6 }{ 4 } }\right \rfloor $$
The graph looks like this with the X-axis being the initial total damage and the Y-axis being the resulting final damage:
Notice that until you reach a damage of 34 the creature will take 0 damage and then at 34 it will take 7 damage. Also note that the end of each line is somewhat misleading. At a value such as 38 which is on two lines, the correct final damage is the higher number. So at 38 the creature would take 8 damage, not 7. Also, yes, the function provided is not exactly the graph shown, I had to make a piecewise so that it was always 0 when the initial damage was 33 or less, but the rest is identical to the formulas given.