Difficulty: Challenge Rating
Easy: APL –1
Challenging: APL +1
Hard: APL +2
Epic: APL +3
In order make an encounter they're "very unlikely to win" you'd want to start at +3 and possibly go to +4 or +5.
What's in a CR?
Taking the big bad guy (or their henchman), calculate their CR either by their Monster entry or by their gear loadout plus Class levels minus 1. Now, fill in the other CR by converting to potential experience. A truly dangerous encounter will involve a control mage and a bunch of minions. Being able to disable PC's is the lifesaver of villains everywhere, and the best way to raise difficulty is to leverage action economy. Throwing twice as many creatures at 1 or 2 lower CR is a much more difficult encounter (reflected by the CR Equivalencies table)
Finally, increase the CR by also putting the PC's in a disadvantaged location/situation. This is one factor that could easily be mitigated by the PC's. Good ideas include "the ranged enemy is at the top of this switchback trail" and "the necromancer cast Unhallow on the area".
The PC's were captured, what now?
Strictly speaking, the DMG suggests that PC's gain experience from "...defeating monsters, overcoming challenges, and completing adventures." However, as @MikeQ indicated, Paizo (and the 3pp who make Adventure Paths) typically award experience any time the PC's survive an encounter. This is prolific in Traps; even if you don't spot it and instead get hit by it, you walked away from the encounter and now know more than you did before you survived a spring loaded axe to the face. Why cause issues, to you as the GM, with planning your next couple adventures' difficulties around if they succeed or not? Give them the experience regardless. It will make your life easier and help the players bandage their hurt feelings when they realize their PC's aren't lost, and in fact are more ready to counter this particular threat when it comes to that.
One more thing
It is surprisingly difficult to knock players out on purpose at levels higher than around 5. Damage quickly escalates over the negative hit point threshold. Make sure to include at least one way of 'saving' a downed PC (such as Breath of Life) and that your minions deal at least some nonlethal damage to increase that knocked out buffer between fighting and death. Consider a focused grappler to take down a pesky character or two with Grapple>Pin>Tie up. It's not fun to be tied up, but losing in general won't be a good experience. This also provides another opportunity for PC's to prepare properly and reduce the difficulty of the fight (in this case, by preparing Freedom of Movement, Liberating Command, and/or Litany of Escape).