It comes down to whichever a DM decides is more specific
We know that specific beats general, but the trouble is knowing which rule is more specific. We know that the standard damage rules are more generic than both the troll's Regenerate feature and, say, a wraith's Life Drain action.
From a wraith's Life Drain action (MM, p. 302):
... its hit point maximum is reduced by an amount equal to the damage taken. This reduction lasts until the target finishes a long rest. The target dies if this effect reduces its hit point maximum to 0.
So if a gang of wraiths jumped a troll and managed to get its maximum HP down to 0, then if a DM decides that the above quote is more specific than (from MM, p. 291):
The troll dies only if it starts its turn with 0 hit points and doesn't regenerate
then the troll dies, otherwise, the troll does not. Even JC has been inconsistent with his rulings on the matter, so it really is up to the DM to decide.
However, if a DM rules that the troll does not die, it likely can't do anything until its maximum HP is restored. (Note that the following is based on rules for players1).
From PHB, p. 197:
Dropping to 0 Hit Points
When you drop to 0 hit points, you either die outright or fall unconscious, as explained in the following sections.
One of which is:
If damage reduces you to 0 hit points and fails to kill you, you fall unconscious (see appendix A). This unconsciousness ends if you regain any hit points.
If the troll does not die outright, then they are unconscious. Since they cannot actually regain hit points, regardless of their Regenerate feature, since (PHB, p. 197):
... A creature's hit points can't exceed its hit point maximum, so any hit points regained in excess of this number are lost.
So the troll will remain unconscious at 0 HP.
1. I know that monsters don't necessarily use the same rules as players, but as far as I am aware, there are no monster rules for what happens at 0 maximum HP. In fact, what little there is on monster HP from the MM, p. 7, implies that the rules I cite are relevant after all:
A monster usually dies or is destroyed when it drops to 0 hit points. For more on hit points, see the Player's Handbook.