Yes, you can, if the Combat Maneuver is made instead of a regular attack.
Look at the descriptions of Combat Maneuvers. Some of them state specific actions, such as Bull Rush, where it says:
You can make a bull rush as a standard action or as part of a charge, in place of the melee attack.
In this case, you can not substitute your AoO for them, as you don't actually get a Standard action (or any other action type). This is for the same reason that you can't cast a spell as an AoO.
Others, specifically Trip, Disarm and Sunder state:
You can attempt to trip your opponent in place of a melee attack.
(or equivalent phrasings). You can use these three as part of an AoO.
Regarding provoking an AoO
Note that even if you attempt a combat maneuver during an AoO, you are not exempt from the usual rules of combat maneuvers provoking an AoO. You can prevent them by having the appropriate Improved ... feat (such as Improved Trip).
Your example
- Enemy leaves a square you are threatening (i.e. can melee attack into), thus provoking an AoO from you.
- You attempt a trip combat maneuver as part of your AoO. This is handled exactly the same as if you did it on your own turn, including provoking an AoO and the potentially resulting penalty on your combat maneuver check (see Performing a Combat Maneuver)
- Your enemy can continue his turn. If you successfully tripped him, the remainder of his move action is lost.