Here's the relevant rules text, described the turned condition:
A turned creature must spend its turns trying to move as far away from you as it can, and it can't willingly move to a space within 30 feet of you.
Clarifying the 2nd Clause
Let's start with the second clause, because it's the confusing part.
it can't willingly move to a space within 30 feet of you
The literal reading implies that any undead within 30 feet of you can't move at all, since all the space around it is within 30 feet of you. That doesn't seem to match the intent of making the undead run away, especially since 30 feet is the range of the spell. I have to conclude that this is just one of the many poorly worded rules in 5e, where the rules as written do not match the intent. That being said, it's not clear what the intent of this clause actually is. I think it's easier to just ignore it, but I've seen two fairly reasonable interpretations:
- The creature cannot end its movement within 30 feet of the caster. This might result in the creature not being able to move at all if it is quite slow, in which case it would just dodge.
- Once the creature is more than 30 feet away from the caster, it cannot move back within 30 feet unless by forced movement, even if the only path to get further away from the caster than its current location would require it to do so.
The second interpretation seems to make the most sense. The undead has to run away, but once it gets at least 30 feet away, it has some freedom in how it proceeds from there. Getting back within 30 feet is just too terrifying, however, so it won't risk it, even if doing so could eventually allow it to get even further away.
Clarifying the 1st Clause
Here's the main part of the condition:
A turned creature must spend its turns trying to move as far away from you as it can
I would look at this not as further at every moment, which would necessarily be a straight line directly away from the cleric or along an obstacle, but as the farthest point from the cleric at the end of the creature's turn.
So, imagine that there is a skeleton 10 feet in front of a cleric, with a wall 10 feet behind the skeleton, at the end of a long hallway that ends in a dead end. The skeleton might run into the corner at the end of the hallway and stop, but it's only about 15 feet from the cleric. Instead, it could run past the cleric and dash down the hallway, getting maybe 50 feet from the cleric. Fifty feet is further than fifteen feet, so that's moving as far away from the cleric as it can.
The second clause doesn't seem to come into play here, although it might in a very large room with one entrance/exit. In that situation, the skeleton might run directly away from the cleric at first (and also away from the door), as that results in it getting as far away as it can on the first turn. Then, after that, it might try to move toward the door, but circling around the cleric at at least 30 feet, too afraid to get any closer.
Since the rule doesn't appear to make sense as written, and we understand that the creature should run away, while trying to avoid getting closer to the cleric, I would do this:
Determine the furthest possible distance the creature can move this turn. This is probably moving and dashing, but could involve other actions depending on the creature and situation. Jumping off a cliff and letting gravity do the work could get the creature very far away, for example.
Compare all the possible locations at which the creature could end its turn, including its current location. Among the available locations, choose the one that is farthest from the cleric. If there is a tie, choose the one that would give the creature the best opportunity to get even further away on its next turn.
The creature moves to the location determined in 2. If the creature didn't move, because there was nowhere better to go, it dodges.
Repeat until the creature is no longer turned.
In step 2, you might add "If the creature is already 30 feet or more from the cleric, don't include paths that require the creature to pass within 30 feet of the cleric." I don't think this is necessary, really, since we're only going to consider paths where the creature ends further away than they started, which was already more than 30 feet away. If they happen to pass within 30 feet of the cleric in the pursuit of getting even further away, no big deal.
If that's too complicated, here's an alternative algorithm, which doesn't really match the rules text. It would instead be summarized by "a turned creature moves as far away from the cleric as possible, and cannot move closer".
If there is no obstacle, the creature moves away from the cleric in a straight line.
Or, if there is a straight or convex obstacle in the way, like a wall or a tree, the creature moves along the obstacle in whichever direction moves them farther away. If the obstacle is at a right angle to the direct line away from the cleric, pick whichever direction gives the shortest path to another escape route. If it's still even (maybe it's the middle of a dead end), just pick left or right at random.
Or, if there is a concave obstacle in the way, like a corner or the inside of a round room, then the creature can't move further away and dodges.
Repeat each turn as long as the creature is turned.
To Answer the Question
As long as they can get further away on their turn, they keep running away. They might get trapped if they aren't very fast and you don't get too close.
In the specific situation of a small room, with creatures with normal move speed, they should run out the door.
If you use the alternative approach, then they would be trapped.