No. No. Mostly no.
One at a time:
Can I apply Sentinel rule 1 to a Battlemaster's Riposte?
No. You can't because the Riposte isn't an opportunity attack. Sentinel rule 1 only works on opportunity attacks.
When a creature misses you with a melee attack, you can use your reaction and expend one superiority die to make a melee weapon attack against the creature. If you hit, you add the superiority die to the attack's damage roll.
Your confusion on this point is understandable. Ripostes and Opportunity attacks are both (usually) melee weapon attacks that use your reaction. But that doesn't mean that every reaction attack is an opportunity attack.
Sentinel (rule 1)
When you hit a creature with an opportunity attack, the creature’s speed becomes 0 for the rest of the turn.
Sentinel strict applies to opportunity attacks, not Ripostes, Ready Actions, or other attacks that you might use your reaction for.
You can make an opportunity attack when a hostile creature that you can see moves out of your reach. To make the opportunity attack, you use your reaction to make one melee attack against the provoking creature.
Opportunity attacks only occur if a creature leaves your reach (or another says otherwise).
If I debuff a creature's speed with Sentinel and trip it, will it be stuck on its next round?
No. The movement debuff lasts for the rest of the turn, not the round (see quote above). A turn is one creature's chance to act. So if a monster moves out of reach during its turn, your opportunity attack also takes place during its turn. As soon as the next creature starts acting, its speed will no longer be 0.
Could I use Sentinel rule 3 and then apply Trip Attack to make the creature prone and have 0 speed?
Mostly no. Again, the attack that Sentinel provides you is not an opportunity attack.
Sentinel (rule 3)
When a creature within 5 feet of you makes an attack against a target other than you (and that target doesn’t have this feat), you can use your reaction to make a melee weapon attack against the attacking creature.
If instead of hitting the rogue, the monster moved out of your reach, then it would be an opportunity attack, and you would be able to trip and debuff their speed. So make sure the rogue knows not to stand right next to you.
Bonus question: Can I use Trip Attack with Sentinel rule 3?
Yes. The only requirement for Trip attack is that you hit with a weapon attack (it doesn't even need to be melee). You can use this with the Attack action, bonus action attacks like Two-weapon fighting, Opportunity Attacks, Sentinel rule 3 attacks or pretty much anything else as long as its a weapon attack.
When you hit a creature with a weapon attack, you can expend one superiority die to attempt to knock the target down...
Different maneuvers have different requirements, so be sure to read each one carefully (for example, sweeping attack requires a melee weapon attack). And remember, only 1 maneuver per attack.