How to balance encounter difficulty in a "live" ecosystem?

Question

I know the headline might seem weird but i couldn't think of anything better.

What I want to do for my next Pathfinder campaign is to have the monsters be as smart and organized as PCs (when their Int Score permits it of course): that means that attacking a goblins nest or bandit hideout without some kind of stealthy approach or killing every enemy that sees you will result in some kind of alert being launched and the whole place coming down to kick you out.

Similarly, failing at a first attempt will make the enemies wary of intruders for at least a few days (e.g. doubling the guards).

How can I balance the encounters / nests so that fights with small groups of enemies (when you maintain a good approach strategy) don't feel too easy (e.g. 1-2 thugs for a 3rd level party), while having the alarm going off doesn't necessarily spell your doom ( e.g. taking on the 10-man gang) ?

What i am searching for is a way to give my player the sense of danger there is in trying to take down an enemy camp that you can't take on all at once, while not automatically killing them if they fail

Answer 1

A living dungeon is similar to a town

That means monsters (or creatures, if you prefer) aren't simply there as XP bags and loot piñatas, they live there, they have jobs or a purpose there, the dungeon has to make some sense without the adventurers, otherwise, it's not an actual place.

So you should have guards if you are expecting to be attacked. Most monster lairs don't have guards, as they expect their home to be well hidden, far from civilization and/or other dangers (from their point of view). Low intelligence monsters shouldn't be very keen on preparing ambushes, traps, alarms or even patrolling their areas. While highly intelligent monsters are another story altogether. They can and should employ every tool at their disposal against intruders.

Guards & Alarms

The amount of guards, how skilled they are, and how quickly they can alert the remaining monsters depends entirely on their abilities and how much technology they have at their disposal. Something simple as arrow slits can help wonders against adventurers kicking down doors and taking you by surprise. Horns and whistles, animals like dogs or loud birds, and even simple traps like metal bars held against a door are enough to alert of invaders (see survival gear).

But one thing to keep in mind is the distance that you want this sound to travel, it doesn't help to alert monsters on the other side of the dungeon if your guards are only at the entrance, far away from where others are, or if the dungeon is full of stone rooms. They will need to actually send a guard to alert the others, and as such, the adventurers might be able to prevent that from happening if they are sneaky and clever.

So, you might need not only guards, but patrols ready to respond to the guards, and alert other monsters of the invasion. They must have a system in place to help the guards and patrols to communicate at long distances so they can be directed when needed, like magical alarms or whistles/horns. Otherwise, they will roam the dungeon looking for the problem instead of going directly towards the problem.

Traps & Hazards

Traps are the easiest way (from the GM perspective) to stop adventurers on their track. Why is that, because every monster (or a good majority of them) should be aware of that trap already, how to avoid it and how to bypass it (if possible). The adventurers don't. Monsters know their homes.

Traps can serve many purposes, from slowing down to alerting of invasions, to harming/weakened and finally, even killing invaders. But you shouldn't rely 100% of them, as a there are always a way or another to bypass traps (jumping a pit, disarming, using divination spells, etc). So organize your traps to work together with your other creatures, a trapped floor may not only drop a careless adventurer to their next xd6 hit point damage, but also fire a loud alarm inside that room, alerting the creatures next door that someone triggered the trap (if their screaming and infighting doesn't, already).

If you are expecting that invaders will pass through a certain room, you can prepare hazards in there, from vines and brambles to slow down their movements, to something more harmful like acid or a nest of snakes, scorpions, spiders or another venomous critter. Since your monsters are aware of that hazard, they will probably think on a way to avoid it, be it an alternative path or a secret door to bypass it completely.

While the invaders are busy slowly getting through that room, the defenders can use that to their advantage and use that time to make ranged attacks or alert the remaining monsters of what's going on.

Time & Location

Keep a good track of where monsters are located, how quickly they can respond to alarms and how fast they can move towards the point where the action is.

Stone walls and medieval doors are heavy, thick, and will block sound. But long corridors and doors with arrow slits or even small entrances to allow sound from passing should be enough to alert whoever is in the next room that there is someone walking in full-plate mail in their direction, or the noises of a spellcaster conjuring detect magic.

So, you will have to consider how far the sound of walking adventurers, conjuring spellcasters, swords swinging against metal and the sound of guards getting killed will travel inside your dungeon (see Perception). And it isn't a bad idea at all to have these things noted dungeon, as you never know when adventurers will be sneaky, or when they decide to be noisy and kick down doors trying to push their way through.

Also, be mindful of their location, you can't possibly fit 10 creatures inside a 3x3 room, one of them will have to stay out. Players (usually) know this, and will try to use this to their advantage. My players, for instance, prefer to fight inside corridors instead of rooms, so they can funnel monsters towards them instead of walking into rooms and possibly getting surrounded. This is a tactic that can also be applied to monsters. If you have lots of ranged attackers, retreat to corridors so they can be protected by the frontline combatants.

Cities and all defense mechanisms to defend them are built by intelligent creatures with a single purpose: survival. So it makes a lot of sense if intelligent creatures also think about their own survival inside their homes.

So, think about your dungeon like a city, it isn't the purpose of any defense to swarm the invaders all at once. They will send a force as an immediate response, then try to counter their invasion using another method. If that doesn't work, they can try to slow them down while they re-organize.

But it's too hard

If monsters are being too difficult for your party to handle, offer them ways to escape (the way they came in), make enemies have expendable resources that they can use too, like healing potions, wands, or even using whistles against them. Many traps are very simple and won't increase the encounter CR by a lot, you may also apply diseases or negative templates (such as young). Afterwall, whoever takes care of the brambles and snakes won't have a very healthy job.

Adjusting combat on the fly is even easier than it sounds. If four players can cause accidents in combat, a dungeon full of loosely-coordinated monsters can cause a lot of havoc in combat. Always remember that they are smart, but they don't share a hive-mind (unless you are fighting formians).

So make area spells affect allies by mistake if players aren't going too well, target that Sleep spell on the elf of the party because you couldn't really tell with all the ruckus and the bad lighting of the place, fire some arrows without the penalty to avoid hitting your own allies and risk hitting them by failing by 5 or less. Some memorable (and laughable) scenes will be born from monsters hitting each other accidentally.

I remember this one time where the boss of an encounter shot a lightning bolt at the ranger of the group, that was hitting everybody from far away with his arrows, and didn't care that his enslaved hieracosphynx was hit in the process. After calculating the damage (I think 9d6), the sphynx dropped to negatives, and the ranger was exactly 10 feet away from the maximum distance of his spell.

Answer 2

I would humbly suggest you ask not 'what would make this fair' but rather 'what would make this fun.'

Years ago, Sid Meier told a story about when they were developing AI for a new Civilization game. They started with the idea of writing the AI to think and act like a human player. The computer player would do things like keeping its units out of the player's field of view, team up with other AI against the player, and when attacking they would do so with overwhelming force. The players' reaction was the AI was being unfair and the experience was not fun.

There's this general appeal of having a living world going on around your character, and this can serve to add a ton of flavor to your campaign, and instigate some really memorable unscripted situations. At the same time, if your players have a goal and they fail or lose or miss out due to an event entirely unknown to them, they probably will not consider that fun.

"After three hours of fighting through minions, you reach the final chamber of the kidnappers lair, but the princess is dead and the kidnappers long gone. Maybe they were tipped off?" It might be realistic, but it doesn't feel very fulfilling. Then again, maybe a gritty realistic medieval simulator is exactly what your players are looking for.

In my campaigns the PCs are the heroes. If it takes braindead enemies and unlikely coincidences to get them there, I will make that sacrifice. I usually to let players miss 'bonuses', but will give them many chances to notice/avoid serious penalties. "The raft's owner refuses to take you across the lake, suspicious of the unnatural fog" "On the way to buy your expensive magical item, you overhear a merchant talking about a glut of magical forgeries." "As you raise your axe to land the killing blow, the orc speaks, 'wait, I know where secret plans are!'"

There is still plenty of opportunity for players to fail. But I think failure should seem fair, otherwise it's not fun. "Now our prize is at the bottom of a haunted lake, I told you we should have listened to the ferryman!"

Answer 3

I've tried this as a GM, and had it tried on me as a player, multiple times and the biggest hurdle in our sessions has always been that players just don't back down. In my experience it's relatively easy to make ad hoc additions or adjustments to small encounters where they need to be more challenging but it is much harder to get PCs to retreat when they're outclassed. I've seen Level 3 or 4 parties try, with complete seriousness, to take on Adult or even older dragons and once an Elder God.

These creatures were only included as plot devices in the first place and it was repeatedly pointed out in no uncertain terms that:

1. you are witnessing this encounter/creature for your characters' education do not mess with it and

2. when they decide to, literally, poke a sleeping dragon, if you pull this stunt you are all Dead.

In all these cases the players were horribly upset by the TPK that resulted, and in one case insisted on trying to go back for round two to the point where the GM threw the game in rather than listen to any more arguments for returning to the dungeon.

As a GM who doesn't particularly like murdering PCs this always left me in a bit of a jam, my solution was this:

• I always aimed encounters low and adjusted them upwards as they progressed until I'd done as much damage as I was comfortable with or, and this always came as a shock, the party thought better of the situation and backed off.

I combined this approach with an "characters advance when I say so" set up so that XP was not directly tracked and it worked quite well.

Depending on your style and how comfortable you are with the Bestiary etc... this approach may not work for you at all, or may be relatively awkward for you to use but with practice it can work.