#Loyalty
DMG p. 93 describes a variant/optional rule for Loyalty. This is intended to be used for friendly NPCs and measuring how far away from betraying/leaving the party they are. It can be used the other way around - how far away they are from betraying their current masters/friends and becoming friends with the party. I have been using it recently for many NPCs and monsters.
You will have to adjucate how the rolls affect the loyalty, and that will certainly depend on the NPC's personality.
For example, a greedy, but brave, NPC might be easily influenced by the party offering him gold, but might get even more hostile by the party trying to intimidate them. Cowardly goblins, on the other hand, might get easily influenced by a good roll in "We will kill you if you don't help us."
Anyway, as far as I remember (especially from PHB, MM and DMG - might have missed XGtE things), there is no "easy" answer for that - it will depend on your (DM) own role-playing as the NPC.
#Why are you fighting?
The more general answer is: you, as the DM, need to understand why is the NPC fighting? Then, it becomes easier to adjucate how actions that decrease that motivation will affect the NPCs. From my earlier examples, if your NPC is a mercenary fighting because he wants money, the party offering him gold will make him a lot less willing to fight. If the NPC fights only due to loyalty, persuading him that his master is a bad guy might work. If a member of his family is held hostage or being threatened, promising protection for that person should work better than threatening the person themselves, as a father might care more for his daughter than for himself.
Create real motivations for your NPCs and that should lead the way for how to role play them.