You can think of marking a foe as paying special attention to that foe and using a fighting style that allows the fighter capitalize on any openings in their defenses. Presumably this combination of attention & stance is sufficiently threatening that most intelligent creatures, and many animals, can recognize the threat it entails (i.e. recognize that taking their attention off the fighter is a good way to get smacked).
Mindless creatures (oozes, plants, some undead, etc.) are incapable of noticing the fighters actions; this doesn't mean that they don't take the attack penalty (the fighter interferes with their attacks) or that they don't trigger combat challenge attacks by attacking the fighter's allies or shifting (the fighter can capitalize on those openings even if the foe doesn't realize the fighter is ready to do so). It may (depending on your DM) mean that those sorts of enemies don't take being marked into account when deciding what to do during combat however (so they might choose to attack the fighter's ally instead of the fighter, even though it's a tactically inferior choice).