Yes, damage from a single attack can roll over
You can tell because the rule says "equal to his total maximum hit points" rather than "equal to or greater than...", which implies that his total maximum hit points are the cap.
In particular, you need to be careful when using merciful technological weapons; the base damage can easily be more than thrice the health of a low-level target. To get around this, consider applying a permanent shrink item effect to your merciful ranged weapon of choice so that you can freely lower the damage dice by four size categories as needed.
Remember that characters don't die unless their hp drop to below negative their constitution score, so even very weak enemies like rats have an extra buffer of close to 10 hit points you can rely on if need be. Just make sure to stabilize them quickly after the attack so they don't bleed out.
(Technically, this isn't true in a strict rules-as-written game, where damage is applied atomically and therefore is only exceedingly rarely equal to a character's total maximum hit points, but that's not the standard reading of the passage.)