This will likely require arbitration and a ruling by the DM
Here's the problem. Specific beats general. Except in this case, you have two specific rules countering one general rule.
The general rule for creature death is in the PHB pg.198:
Most DMs have a monster die the instant it drops to 0 hit points, rather than having it fall unconscious and make death saving throws.
Mighty villains and special nonplayer characters are common exceptions; the DM might have them fall unconscious and follow the same rules as player characters.
The exceptions, which are specific overrides to this are Instant Death and Knocking a Creature Out.
You've already cited both of these sources in your question, so I'll break it down.
Instant Death - specific rule that occurs if and only if maximum damage is applied to a character/creature that exceeds it's maximum health, taking into account damage required to reduce it to an unconscious state. This means a target with 9 HP left out of 15 HP total would require a total of 24 damage to be dealt in a single turn in order to kill it outright.
Knocking a Creature Out - specific rule that occurs upon dropping a target to 0 HP. You can elect to deal this nonlethal damage on the instance of attack.
Now, why is this a DM decision? Because even a nonlethal attack can end up becoming a lethal attack by accident. Let's say you take a swing at a very wounded goblin and declare nonlethal intent to knock the creature out. So you roll your attack and score a critical hit.... and you're playing a Half-Orc so you also have Savage Attacker bolstering your blow.
The damage you deal is reflective of the attack that has been made. Which means a critical, nonlethal blow means you screwed up. For instance, I would narrate that as, "You hit the target in the head with the flat of your blade to knock it out, but in it's weakened state you snapped it's neck with the force of your blow." (Or something similar among those lines, like internal bleeding)
Neither of these specific rules take precedent over the other though. So you could also rule as the DM that the nonlethal intent is always successful and thus the damage is irrelevant so long as it is enough to render a target unconscious.
What this boils down to is a rare case where two specific rules conflict, so arbitration becomes a requirement if there is disagreement. Ultimately, rolling high/low might be the easiest way to determine the outcome when torn between two choices.