The spell Life Transference (from Xanathar's Guide to Everything) states that you can heal a creature equal to twice the damage you self-inflict by the spell.
If a half-orc uses this spell while on very low health and activates its Relentless Endurance racial ability, will it heal based on the total damage or only the initial HP - 1?
Is there any difference if I don't use my Relentless Endurance?
Relentless Endurance states:
Relentless Endurance. When you are reduced to 0 hit points but not killed outright, you can drop to 1 hit point instead. You can’t use this feature again until you finish a long rest.
Since the spell says nothing about killing you outright if this damage reduces you to 0 hit points, and there's no mention of the source of the damage, yes, the ability would activate when you take damage from Life Transference, which states the following:
You sacrifice some of your health to mend another creature’s injuries. You take 4d8 necrotic damage, and one creature of your choice that you can see within range regains a number of hit points equal to twice the necrotic damage you take.
There is no mention of something along the lines of "this damage can't be reduced in any way", as the Wish spell does:
Wish. [...] The stress of casting this spell to produce any effect other than duplicating another spell weakens you. After enduring that stress, each time you cast a spell until you finish a long rest, you take 1d10 necrotic damage per level of that spell. This damage can’t be reduced or prevented in any way.
Therefore, since spells in D&D 5e only do what they say, yes, you can use the spell in this way (at least according to RAW - your DM might decide otherwise).
To answer your question about how much healing is applied, this is what happens:
You cast the spell -> you take 4d8 damage + the creature of your choice within range regains the exact same amount -> you are reduced to 0 HP -> your ability activates, so you have 1 HP instead.
There's a difference between the amount damage you take, and the amount that your hit points are effectively reduced. On page 196 of the PHB, the section Hit Points states:
A creature’s current hit points (usually just called hit points) can be any number from the creature’s hit point maximum down to 0.
Therefore, if you're e.g. at 10HP, cast the spell and roll 17 damage on the 4d8, you would be at -7 HP. However, your hit points can only be between 0 and max, so you drop to 0 instead - but you still took 17 damage.
The difference between damage taken and HP lost is further supported by the section about instant death, PHB p. 197:
Instant Death Massive damage can kill you instantly. When damage reduces you to 0 hit points and there is damage remaining, you die if the remaining damage equals or exceeds your hit point maximum.
