Only Gentry and Nobles Ride Horses!!!
The primary reason PC's would walk instead of ride would be that they are not nobles nor elves. In most realistic settings, a horse is a status symbol of the gentry and nobility.
In many real world timeframes, a non-noble on horse was subject to harsh, sometimes even capital, punishments. As in, many lashes or even breaking on the rack or being drawn and quartered.
So, if they rode, it would be oxen, mules or donkeys. Which means, no faster than walking anyway, due to forage, slow pace, and stubborn animals.
Any muddy ground is a problem
Equines are not happy when they can't keep their hooves relatively dry. As in, able to keep the hoof above the surface most of the time. March them through swamps, bogs, and other such nasty, uneven and non-visible bottomed terrains, and you either have lame horses or nearly lame horses. A horse doesn't mind getting wet, per se, but the do need to keep the frog from developing yeast infections and the hoof from tannic acid softening. And that does mean drying off after getting wet, and not staying wet for days on end.
Unsteady and uneven Ground is a problem
Horses generally dislike ground thatwill give way under them: cliffs, barrows, weak-dome soon-to-be-sinkholes, gopher or mole burrowed, or loose soils on hills. All are great ways fora horse to break a leg.
Likewise, off-trail woods and mountains present real risks due to stable but uneven footing.