If the PCs lose a fight but do not die (e.g. imprisoned), do they gain experience points from the battle?
Only from the monsters they defeated.
You gain experience when you defeat a monster (emphasis mine)
When adventurers defeat one or more monsters - typically by killing, routing, or capturing them - they divide the total XP value of the monsters evenly among themselves.
[Dungeon Master's Guide page 260]
Even if you lose a fight, any monster you defeated during the fight would grant experience. The ones you don't defeat (i.e. the ones who imprisoned the party) would not.
The challenge isn't always just combat
Sometimes other factors provide XP, which will also be awarded if the player's defeat that aspect of the encounter.
Some examples are:
- Traps and/or hazards that give XP
- Encounters where "defeating" the monsters involves escaping rather than beating them in combat (or some other challenge)
- Social encounters that qualify as a sufficient challenge
The DM has quite a bit of freedom in the awarding of experience.
First I'd take care of the obvious:
- If they subdued or killed any of the opposing group, award experience.
Now, the more subjective awards:
- How did they end up imprisoned? Did they talk the opposing group out of killing them? It's a bit of a stretch, but if they took any action that resulted in them not-dying then I would give them something. Not a lot, but a token gesture.
Personally, I would give them experience for any monsters they killed, but not for the fight itself. If they were overpowered and imprisoned it may because they aren’t strong enough for the monsters your throwing at them, and they need more experience to put into leveling up.