Yes, you can use the ability.
Relentless Endurance states:
Relentless Endurance. When you are reduced to
0 hit points but not killed outright, you can drop to 1 hit
point instead. You can’t use this feature again until you
finish a long rest.
Since the spell says nothing about killing you outright if this damage reduces you to 0 hit points, and there's no mention of the source of the damage, yes, the ability would activate when you take damage from Life Transference, which states the following:
You sacrifice some of your health to mend another creature’s injuries. You take 4d8 necrotic damage, and one
creature of your choice that you can see within range regains
a number of hit points equal to twice the necrotic
damage you take.
There is no mention of something along the lines of "this damage can't be reduced in any way", as the Wish spell does:
Wish. [...] The stress of casting this spell to produce any effect
other than duplicating another spell weakens you. After
enduring that stress, each time you cast a spell until
you finish a long rest, you take 1d10 necrotic damage
per level of that spell. This damage can’t be reduced or
prevented in any way.
Therefore, since spells in D&D 5e only do what they say, yes, you can use the spell in this way (at least according to RAW - your DM might decide otherwise).
The creature heals whatever you rolled on your 4d8 (or more, depending on the spell level).
To answer your question about how much healing is applied, this is what happens:
You cast the spell -> you take 4d8 damage + the creature of your choice within range regains the exact same amount -> you are reduced to 0 HP -> your ability activates, so you have 1 HP instead.
There's a difference between the amount damage you take, and the amount that your hit points are effectively reduced.
On page 196 of the PHB, the section Hit Points states:
A creature’s current hit points (usually just called hit
points) can be any number from the creature’s hit point
maximum down to 0.
Therefore, if you're e.g. at 10HP, cast the spell and roll 17 damage on the 4d8, you would be at -7 HP. However, your hit points can only be between 0 and max, so you drop to 0 instead - but you still took 17 damage.
The difference between damage taken and HP lost is further supported by the section about instant death, PHB p. 197:
Massive damage can kill you instantly. When damage
reduces you to 0 hit points and there is damage
remaining, you die if the remaining damage equals
or exceeds your hit point maximum.