How to handle prepared actions when you are expecting to see an enemy but you haven't yet?

Question

In my last session my group and I have encountered this situation:

We found a trap that would trigger two hostile creatures to enter our room through a portal each. We knew that because there was a very similar setup previously. Our Bard failed to disarm the trap so we decided to trigger it and wait for the enemies and ambush them. My dwarf fighter strode on and positioned himself near one portal and declared to prepare the action "when any enemy exits the portal I will strike him".

Only at this point, when two demons appeared through the portal, our GM asked us to roll the initiative.

Was my prepared action triggered and I would strike before the initiative began, or it was lost just because I prepared it when the initiative wasn't yet rolled?

Answer 1

Rolling initiative is how you adjudicate out-of-combat preparation.

When do you ask players to roll initiative? In most cases, it’s pretty simple: you call for the roll as soon as one participant intends to attack (or issue a challenge, draw a weapon, cast a preparatory spell, start a social encounter such as a debate, or otherwise begin to use an action that their foes can’t help but notice). (source)

There's no explicit rule for whether you can ready an action outside of combat (although my reading would be "you can't"), so look at how it would play out: you know the demon is coming through the portal; the demon knows that someone triggered the trap. Both you and the demon would prepare yourselves to engage with the enemy once they're visible and attackable. How do you decide which character has a quicker reaction to their opponent becoming visible? Roll initiative.

Transitioning from exploration to an encounter usually involves rolling for initiative. Call for initiative once a trap is triggered, as soon as two opposing groups come into contact, or when a creature on one side decides to take action against the other. For example:

• A group of PCs are exploring a cavern. They enter a narrow passage patrolled by a group of kobold warriors. Now that the two groups are in the same area, it’s time to roll initiative.
• Merisiel and Kyra are negotiating with the kobold king. Things aren’t going well, so Merisiel decides to launch a surprise attack. As soon as she says this is her plan, you call for initiative.

(source)

If readying an action worked out of combat, surprise attacks would always be freebies: just define the trigger as "the king says something." Prior to the demon coming through the portal, neither you nor the demon can see each other. The advantage you have is that you could be anywhere in the room, while the demon is coming out through a fixed location (presumably), giving you more leeway to be sneaky:

When one or both sides of an impending battle are being stealthy, you’ll need to deal with the impacts of Stealth on the start of the encounter. Anyone who’s Avoiding Notice should attempt a Stealth check for their initiative. All the normal bonuses and penalties apply, including any bonus for having cover. You can give them the option to roll Perception instead, but if they do they forsake their Stealth and are definitely going to be detected.

To determine whether someone is undetected by other participants in the encounter, you still compare their Stealth check for initiative to the Perception DC of their enemies. They’re undetected by anyone whose DC they meet or exceed. So what do you do if someone rolls better than everyone else on initiative, but all their foes beat their Perception DC? Well, all the enemies are undetected, but not unnoticed. That means the participant who rolled high still knows someone is around, and can start moving about, Seeking, and otherwise preparing to fight. The characters Avoiding Notice still have a significant advantage, since that character needs to spend actions and attempt additional checks in order to find them. What if both sides are sneaking about? They might just sneak past each other entirely, or they might suddenly run into one another if they’re heading into the same location. (source)

If you can hide to the side of the portal and the demon doesn't immediately notice you, you'll be undetected and have the advantage. If you aren't hiding (or otherwise preventing the demon from recognizing you as a threat), you and the demon will see each other at the same time and roll initiative.

Ultimately, allowing readying an action outside of initiative order undermines the purpose of initiative and can lead to outlandish results like opening a door and having all six enemies immediately getting to make an attack on you because they knew you were coming. The system for adjudicating who gets to act first once both parties are on equal ground and can see each other is rolling initiative.

Answer 2

You likely roll for initiative, and certainly if the DM says so

In general your DM is right, you would roll initiative at the beginning of an encounter, and no attacks should be made before initiative. But there is an exception. The Core Rulebook pg. 498 has this option:

Initiative After Reactions
In some cases, a trap or a foe has a reaction that tells you to roll initiative. For instance, a complex trap that’s triggered might make an attack with its reaction before the initiative order begins. In these cases, resolve all the results of the reaction before calling for initiative rolls.

How would these foes set up their reaction to attack you? And conversely how could the characters, if they are the "foes" that set up an ambush for some other monster? There are special monsters that explicitly state they have reactions that happen before initative, like the hunting spider. The only way for the characters, who lack such special reactions, would be to use the Ready action, which sets up such a reaction. But is it possible to take that action outside of combat?

The rule on Actions on p. 461 Core Rules states that you can can use actions outside of encounter mode:

You affect the world around you primarily by using actions, which produce effects. Actions are most closely measured and restricted during the encounter mode of play, but even when it isn’t important for you to keep strict track of actions, they remain the way in which you interact with the game world.

Unfortunately, the Ready action talks about doing something outside your turn, which would imply that you already are in encounter mode and turn order, because a turn is only defined once your are under initiative order. From the Key Terms section for Turn:

During the course of a round, each creature takes a single turn according to initiative.

So this would rule out that you can ready an action on technical grounds.

While there are rules for surprise, they primarily deal with how you can use stealth to get a leg up on initiative. In this situation, the summoned creatures have no way to discover the characters before they appear, and therefore the rules for Stealth and surprise at the start of an encounter do not make much sense.

I think one way to resolve these contradictions between the Ready action and the Initiative after Reactions rule that makes narrative sense is to allow characters to Ready an action outside of combat, if the circumstances are right, and hold it until the start of the encounter, where it could resolve before rolling initiative.

So in your situation you could attack with your readied action, and then roll initiative. But because technically you cannot Ready an action before you are in initiative, in the end it is your DMs decision how they handle it. By default, you would have to roll intiative.