I was just thinking about this problem the other day. The solution that I allowed myself to accept was based on the idea of experience. These heroes are interacting with characters and challenges that make them more experienced over time, allowing them to overcome greater challenges in the future. I will give two possible solutions, the latter of which is actually a very old solution.
One is that you treat every encounter and challenge as a single instance of experience. While you might gain say, social acumen, by growing up with your family. There is only so much you can get by overcoming the same 'encounter' over and over again. If you left the nest thinking yourself to be a social expert because you've won the argument with your sister over how to divide the tv schedule for 100th time, you will quickly find that it didn't prepare you for the outside world. An encounter, once it has entered into some form of completion where the obstacle that encounter represented is no longer there, can give experience once (whether you want the experience to be awarded after the first instance, or at the end of the process is up to you, note that if players are tallying their EXP after every combat, they may suspect that the NPC's they scared away might come back if you don't award them the experience for that combat).
Based on this interpretation, I would consider the listed EXP on a monster to be all the experience they can provide to an individual over a lifetime, whether it is awarded all at once or not, whether it is awarded for combat or not.
A second possible measure is using some sort of gauge as to the success of the players as adventurers. Did the players get something out of finishing the combat or solving the puzzle? Sometimes that reward is not dying, at other times it is getting some favor or a sum of gold. Like the previous example, I can provide an irl justification for this method. If someone is very determined in their job to get more customers, and tries 30 solutions to do so, it is very likely that this person's understanding of the issue was fundamentally wrong. If experience is in fact practical wisdom applied to achieve a goal, then the achievement of goals itself is a direct measure of a character's experience.
For adventurers, a metric to consider would be the acquisition of gold - whether or not the character decides to take it or not. This is the method of gaining experience in Advanced Dungeons and Dragons, 1st Edition. If the players scared the goblins away, but aren't able to save the prisoner or find the treasure chest because the threat of the goblins are still there, and they've blocked up the passage - have the characters really learned something from the experience? (plenty of people would say yes, no worries, you wont offend me by disagreeing).
The solution I would propose is not a direct gold to EXP as was present, but rather awarding EXP for the encounter when the 'dungeon' or 'area' is finished and the players have their reward. The value of the 'area' would be equal to the total exp of the monsters and challenges within, perhaps with some bonus from miscellaneous goals or treasure. If they can talk their way through the goblins and stealth the goals and gold away from the hideout, then I would award the full exp value of the dungeon. However, if they are missing something, I would decrease proportionally.
Having tried this in the past, I settled on making a list of things present in the dungeon (treasure, plot hooks, macguffins, favors, information and other rewards), and giving players a letter grade based on how many they found - then rewarding that percentage of the dungeon experience (Complete=100% A=90%, B=80%, C=70%, D=60%, F=50%, E=30%, Abandoned=0%).
On a personal note, I am currently running a combination of Lost Mines of Phandelver and Keep on the Borderlands (where the Cave of the Unknown = the Cragmaw Hideout), I have been very simply awarding 1 experience point for each gold piece and experience points for each monster they kill or 'resolve'. Resolve meaning that that monster is no longer relevant in any way to the game, and wont come back, appear, or even be present if the players choose to come back to an old dungeon. In all 3 cases, I would award EXP for those goblins only once.
I hope that helps!
