I'm running my first real campaign. The idea I have for the first dungeon involves the PCs coming upon two sides already engaged in combat. One side is losing and weaker than the other, though not in an altogether different CR range, but both are enemies of the PCs. How do I calculate the encounter difficulty?
An "encounter" isn't necessarily "battle". There are multiple ways to succeed in an encounter, and often are the non-combat ways the most flavorful. One can also combine different approaches with battle to place themselves into an advantageous position. In this scenario, you should balance the encounter for the way you wish it to be resolved.
If you want your PC's to jump into the fray and take both groups on, you'll make it so that having all enemies fight the PC's exclusively would be on the high end of Hard, or the low end of Deadly. As the fight progresses, you have your PC's targeted depending on how much attention they're drawing. You throw in some NPC on NPC combat to adjust the difficulty whenever the fight starts going really bad for the players. You don't have to strictly simulate that combat. Instead just say "During this round, the first side lost X people while the second side lost Y people. The others that aren't attacking you are still in a furious battle.", or "As you're about to be killed, your enemy gets distracted by a sword swinging towards his head from his side. He instead turns to defend."
If you want your PC's to take an alternative course of action, you should make the encounter dip well into Deadly (again balancing for the case that all NPC's would attack only the players). This would discourage making themselves targets of all the available enemies. The players could instead try to just sneak past the fight, stealthily thin down both sides (reducing their numbers) before entering the fray, or ally themselves with one side. They could even attempt peace negotiations, in which case they would likely be ignored, unless a few enemies from either side break off from the main fight to get rid of the nuisances. Or they can just wait in the shadows until one side wins, and then execute the remainder. The encounter that results from a smart choice doesn't even have to be Hard. Intelligence itself should be rewarded.
Either way you choose to balance the encounter, make sure to determine different XP rewards based on the results. The different results possible are: avoid fight, help the weaker side, help the stronger side, or kill all. Make sure that, in the case of helping one side win, they get some extra XP for the tactical decisions made and the Charisma checks needed to establish the alliance. You would add that to the count of dead bodies on the other side. Keep in mind that allying with the stronger side might be harder for your players to do as that side has slightly less incentive to comply. For the losing side, a simple "don't kill us and we'll help you" could suffice.
In the case the players wait for one side to lose before they jump in, just decide what a regular two-sided encounter would be, and say that's how many enemies survived. You can design the new encounter before the session the same way that you would make any other simple two-sided encounter. Throw in some extra XP for doing the smart thing if you want.
It's trivial to design an encounter that is difficult if the two factions (the two NPCs) join forces to take on one stronger faction (the PCs), but that is also able to provide a challenge if they started turning on each other.
I recently designed a similar encounter and pitted it against 2 5th-level PCs, a good starting point is to think of the the worst-case scenario first.
The PCs done screwed up; through their in/actions earlier in the game or before the fight, they have managed to earn the ire of both factions enough that they set aside their differences to team up against them.
Calculate the encounter as Deadly or Hard. Assume the two factions work harmoniously with each other (at least until after all the PCs are dead). As an example, the encounter I designed with 2 5th-level characters in mind had six CR1/2 enemies, a Hard encounter for only 2 PCs of their level, but in my experience, certainly beatable.
Using clever tactics, wit and/or fast-talking, the PCs manage to rouse up the two factions' hatred for each other, and it becomes a big messy free-for-all.
After designing a tough but beatable encounter above (see worst-case scenario), you really can't go wrong after that, and can simply decide- in my case, arbitrarily, how many NPCs you want to attack the PCs.
Take for example the 2 5th-level PC scenario, if you pit five NPCs against the PCs, it's a Medium encounter, if you have them fight only half of them, three, it's Easy.
TL;DR Design the encounter assuming both factions attack the PCs, and slide down, taking out NPCs off the calculation as needed.
I've used this in my games, and have had both scenarios come up, and both fights seemed to have been fun and satisfying.
To handle any encounter with moving parts, you have to decide how you're going to handle it. Will the two enemies continue fighting each other, or will they both set their differences aside and turn on the player(s)? Or will it be an every-man-for-himself fight where people are just going berserk? What I usually do/run into is have an every-man-for-himself, and go turn-by-turn to decide who is attacking who, and then simulate damage, and only count experience based on the number of enemies killed by the player(s) as opposed to all enemies killed. You might also have good use of an encounter calculator.
Honestly, the only thing I could think of that hasn't been already said would be make a small piece of each opposite battling party attack the PC's. For example. 8 Kobolds are fighting a group of 4 Gnoll. The party attacks (if thats what their choice is) and 3 Kobolds and a Gnoll attack the party. The remainder of the mobs continue to battle each other. Then if one side or the other wins before the PC's kill the mobs that attacked them, they join the fight against the PC's or the minority mob, then the PC's.
It's how a real fight would be. just because the two fighting parties of mobs see the PC's or are engaged by the PC's doesn't mean that their battle ends. They stop fighting one side, and it doesn't stop fighting,the battles over.. It also depends on what the mobs happened to be fighting over as well. Some rolls to decide if they hate the opposing mob more than the PC's might be in order as well.
It also depends on if the PC's attack right awy or if they're going to wait and see if the Mobs kill each other off. There happens to be a ton of variables in this question
Quick adjudication off the top of my head:
There are three fights, PC v G1, PC v G2, G1 v G2
The PC's are dealing with 2/3 of the fights, while G1 v G2 is only one leg.
2*(CR(G1) + CR(G2)) / 3
but thats just to keep it easy. I am sure some in depth analysis would wield vastly different ideas.