The spell is more specific, so the troll dies
The troll's regeneration ability is a specific rule that overrides the general rule, which is that monsters die immediately upon dropping to 0 HP. It's telling you that simply dropping the troll to 0 HP does not kill it, and dealing damage to the troll while it is at 0 HP also does not kill it (since the troll has no death saves to fail unless the DM decides otherwise). In essence, the ability modifies the normal rules for monster death. The troll's death from being at 0 HP is delayed until after the start of its next turn.
Spells like power word kill bypass the normal rules for creature death entirely, so they also bypass the modified death rules given in the regeneration feature. Power word kill kills any creature at 50 HP even though creatures do not normally die until they reach 0 HP, because the spell says it does. Power word kill also kills a troll at 50 HP (or 0 HP, etc.) for the same reason.
In summary, instant death effects bypass normal rules for death entirely, which means the modifications to the normal death rules made byin the regeneration feature are rendered moot.