In these specific cases, the rules are explicit.
Open Hand happens before hellish rebuke.
Since hellish rebuke triggers when you are damaged and Open Hand triggers when you are hit before any damage is dealt, Open Hand unambiguously triggers first.
Shield happens before Open Hand.
See Darth Pseudonym's answer for the passage in the rules that reinforces that shield is intended to occur first. This passage isn't in the text of the spell, which is problematic, but it is there in the rules and does resolve the ambiguity. I won't replicate Darth's answer here.
In the general case, there is an optional rule.
Notably, hellish rebuke and shield aren't the only ambiguous situations that might ever come up when using Open Hand, and there is no core rule covering them all, but, the problem of simultaneous timings comes up often enough that an optional rule was released in Xanathar's Guide to Everything, in the aptly named section on Simultaneous Effects:
If two or more things happen at the same time on a character or monster’s turn, the person at the game table — whether player or DM — who controls that creature decides the order in which those things happen.
If you use this optional rule consistently, then the player of the monk (whose turn it must be, since they just used Flurry of Blows with a bonus action) can decide whether its Open Hand effect triggers first or whether its target's shield spell triggers first.
It's likely the monk will choose to trigger its effect first (in which case the target no longer has their reaction to use in order to follow the hit up with casting shield), unless it wants to compel the target to spend a spell slot (in which case the shield spell prevents the hit from occurring and the monk still tactically benefits from the result). Regardless, the decision would be up to the player of the monk.
Using this optional rule fits nicely with a DM's intuition that Open Hand is more specific than shield and that the monk should be allowed the chance to shine in this way. It's also pretty clear that the intention of Open Hand is to provide a way to rob a target of a reaction, and this ruling elevates that intent. As a bonus, this ruling also works for shocking grasp and other features with similar timing.
This ruling violates the stated intent for shield in the rules, as explained in the previous section, but my goal is to teach you to fish rather than giving you a particular fish. Ideally, knowing about this optional ruling will help you resolve future issues of ambiguity with Open Hand.