What is the mechanism to dismount an opponent in mounted combat? What are the rolls?
The only rules for an "unseat" action I've found are the unseat feat, which enables a lance-wielding rider to get a free bullrush attempt against another rider. If successful, the other rider automatically falls off his mount and prone.
While not tripping, there are various things that could work as they do allow you to move opponents in various ways. It might however come down to DM variance.
Drag: Grabbing the target and hauling him out of the saddle by main force(straight backward)
Bull-Rush: May need a jump check(Flying tackle)
Reposition: Drag an opponent around(with the exception that it must remain remain within reach)
There is also the feat Pushing Assault(APG)(you need power attack, STR 15, BAB 1 and a two handed weapon as pre-reqs) which lets you push an opponent backward.
There are also the monster abilities Pull and Push that do much the same as Drag/Bull-Rush(with the limit of it must be a smaller creature than the monster)
The trip maneuver inflicts the prone condition. I would think that involves being off the mount on the ground.
The NPC Codex tactics are a bit notoriously not always rules correct, but the Horse Monk from that book has this as his tactics:
During Combat The monk prefers to fight from horseback, and uses Ride-By Attack and Lunge to hit foes so he faces little risk of retaliation. He often uses Stunning Fist on the first pass, and if successful follows with a Spirited Charge. Against other mounted foes, he uses Improved Trip to unhorse his opponents if possible.
The Unseat feat seems to imply that a Bull Rush does the same thing (which would make sense, if you get pushed 5' you're not on your horse...) I would think a rider would have some bonuses against both maneuvers based on higher ground and type of saddle...
This might seem a bit obvious, but killing the mount usually works. Most mounts encountered in play are horses, and most horses don't have more than the standard hit dice for their kind, so a sneak attack or damaging spell can reliably kill a mount in one shot after the first few levels of play.
If you need or want to keep the mount alive, nonlethal damage sneak attacks are fairly easy to arrange, and there's a wide variety of save-or-X spells that target any given mount's weak saves.
The Ride skill says the rider needs to make a check when they take damage under the Stay In Saddle special action.
Stay in Saddle: You can react instantly to try to avoid falling when your mount rears or bolts unexpectedly or when you take damage. This usage does not take an action.
It is only a DC5 Ride Check but you could add situational modifiers which might be applicable. I'm a little surprised that there isn't the same mechanic as spellcasting in combat has, where when the spellcaster takes damage the DC is raised by the amount of damage taken. Personally I would houserule that to be the same for riders so taking 12 damage would need a Ride DC17 check to stay mounted.
RAW the rider needs to pass a DC5 Ride skill check to stay mounted when they take any damage.
