Starting an encounter is never spelled out explicitly nor are there any concrete examples I'm aware of. However, you can piece together rules on Hide/Spot, Move Silently/Listen, Stealth and Detection in a [environment], Surprise and How Combat Works to come up with a method something like:
First, determine if anyone is hiding. If no one is hiding, you can skip to step 3 of How Combat Works when line of sight is obtained.
Determine if Spot or Listen is possible and if so at what distance based on environment. Walls or obstructions may prevent Spot and hinder Listen, while the absence of cover or concealment prevents Hide.
Roll Hide/Spot Move/Listen for everyone involved and add modifiers for distance. Technically, you could roll this over and over until someone is aware or you could just modify the distance until at least someone is aware based on one set of rolls, provided at least one side is closing.
If at least one creature is aware from each side, and at least one creature is unaware, a surprise round ensues. Until then, aware creatures can act freely, or start the surprise round by attacking.
Freely, with the caveat that anything that alerts the other side is going to change the underlying premise; that only one side is aware, thus a surprise round ensues. If you roll initiative while only one side is aware, that's OK, but you're going to have the possibility of multiple surprise rounds, which may be awkward.
The three previous paragraphs are implicitly mentioned in step 2 of how combat works.
How Combat Works
Combat is cyclical; everybody acts in turn in a regular cycle of rounds. Combat follows this sequence:
- Each combatant starts out flat-footed. Once a combatant acts, he or she is no longer flat-footed.
- Determine which characters are aware of their opponents at the start of the battle. If some but not all of the combatants are aware of their opponents, a surprise round happens before regular rounds of combat begin. The combatants who are aware of the opponents can act in the surprise round, so they roll for initiative. In initiative order (highest to lowest), combatants who started the battle aware of their opponents each take one action (either a standard action or a move action) during the surprise round. Combatants who were unaware do not get to act in the surprise round. If no one or everyone starts the battle aware, there is no surprise round.
- Combatants who have not yet rolled initiative do so. All combatants are now ready to begin their first regular round of combat.
- Combatants act in initiative order (highest to lowest).
- When everyone has had a turn, the combatant with the highest initiative acts again, and steps 4 and 5 repeat until combat ends.
If the environment is a dungeon, due to additional Listen modifiers from walls and doors and the inability to Spot due to lack of line of sight, encounter distances are typically closer than outdoors and generally creatures will hear before they see enemies. Outdoors, encounters can start much further, depending on the terrain. If there is no cover or concealment, or no one hears anyone else, then once line of sight is obtained, you can just skip to step 3.
Do you assume that the parties are moving directly towards one another, or does travel direction not play any part under RAW rules?
The direction the party is moving is up to the party, and the direction of the enemy up to the DM (you).
In other words, once you have reached the max distance from the DMG, how does a spot and/or listen check determine distance?
Spot and Listen does not determine distance, rather, a roll based on environment (or better, your judgement based on terrain features or lack thereof) determines the maximum distance at which two opposing groups have a chance of spotting a hidden opponent (Spot/Hide). Beyond this, the groups can't see each other. If neither group see each other at the maximum distance and at least one group is closing distance, the distance modifiers will change as they move and eventually someone will see an enemy.