In general, no, but massive damage is a weird undecidable case.
The rules support Relentless Endurance both applying and not applying in this case.
To die by massive damage, the damage must reduce you to zero hit points:
Massive damage can kill you instantly. When damage reduces you to 0 hit points and there is damage remaining, you die if the remaining damage equals or exceeds your hit point maximum.
(Separately, you can die by massive damage if you're at zero hit points already, but we're not considering that case.)
If you assume Relentless Endurance applies, it prevents you from being reduced to 0 hit points. You drop to 1 hit point instead of 0, which means the damage doesn't reduce you to 0 hit points, and we don't even check whether there's enough leftover damage to kill you outright. You aren't killed outright, so Relentless Endurance is indeed allowed to apply.
On the other hand, if you assume that Relentless Endurance doesn't apply, then you do drop to 0 hit points and are killed outright, and thus Relentless Endurance shouldn't apply.
(Even if Relentless Endurance does protect against this, there are causes of instant death that will get around it, such as power word: kill, getting Strength-drained by a Shadow, or having your maximum HP reduced to 0 by a Wraith. So neither ruling would completely negate any part of the rules.)