This is a generalization of the rule "Don't make your players roll unless there is a consequence to failure and a reward for success." You need to consider the effect of the character moving outside of combat, and whether their movements are reasonably likely to affect the combat. Luke's comment that "If someone across the continent is scrying on, communicating with, or teleporting to a combatant, they probably need to abide by turn rules too." is spot on.
The answer, then, is that this has nothing to do with combat per se but about how important the exact flow of time is to the situation. If the PCs are camped out in the middle of nowhere with no apparent hostiles, but they need to do X, Y, and Z within ten minutes or else their food will spoil, then time matters as much as if they were exchanging blows with orcs.
Let's consider the effect of this outside of apparent, immediate combat scenarios.
Scenario 1: The PCs are in town, and they decide to visit the butcher, the baker, and the candlestick maker for various reasons because story. The Big Bad is sitting in his castle's study gloating about how invincible he is and drinking goblets of mead, and nothing is likely to change with regards to that in the next few hours. There's no apparent consequence for taking one route around town or another, or lingering in Town Square for an extra few minutes gawking at locals, so there's no need to track things turn-by-turn. We assume that any reasonable shopping trip will conclude before future world-shaking events transpire.
Scenario 2: The PCs are in town, and they decide to visit the butcher, the baker, and the candlestick maker for various reasons because story. They are being hunted down by the Town Guard, invading orcs, "enforcers" from the local Thieves' Guild, blink dogs, drunken thugs, their family, whatever (take your pick). In this case, even though combat is not happening, you need to understand where people are going and how fast they are going in order to determine when or if they will meet.
Scenario 3: The PCs are in town, and they decide to visit the butcher, the baker, and the candlestick maker in order to gather the supplies necessary to repair the local dam, which will burst in five minutes. In this case, you probably do need to track the PCs around town (maybe they can hire a cab, or try a Knowledge check to determine if they know of any back-alley routes?) and determine how many turns it takes to complete the transactions. You could even have fun with this - do the PCs want to pay "sticker price" now, or spend precious turns trying to haggle the merchant down, or even trying to steal? Time is of the essence, but not paying full price is still awesome...