Two rules observations, then a more extended answer:
The Character Advancement section on page 30 says, in part,
The rate of this advancement depends on the type of game that your group wants to play. [...] In the end, it is up to your group to decide what rate fits you best.
So, you're absolutely within your rights to work with your group to find a satisfying pacing.
Additionally, there are certainly some downtime activities available in Pathfinder. However, they're not inherently interesting. They may lead to interest; "to craft this item, I need to sleep in the Cave of Dreams for a night", but they're not interesting on their own.
Time as a Limited Resource
Constraints make for interesting stories and hard choices. Suppose Fred the level 1 fighter just stumbled on the dragon's lair, walked in, and killed it in one punch. Not exactly literary gold.
On the other hand, imagine Fred the Fighter arrives just in time to see the burglar (who has been terrorizing the town for months now) backstab Claire the Cleric and run away. Fred could chase down the elusive thief and leave Claire to possibly die, or he could administer a healing potion and certainly lose track of the thief. The time constraint inherently adds drama to the situation.
Now consider that on a grander scale. Instead of rounds, think days or weeks. You could spend a few days tracking the roving bandits in the foothills, but you may not find them. Or, you could spend a week guarding an important caravan to the next major town. Either way, you'd better be back in a week and a half when the harvest finishes, or the Gnoll tribe will likely steal next year's seed corn.
Think bigger. The 14 year old crown prince can't rightfully claim his throne until he's turned 16. Of course the neighboring kingdom is sending assassins in the hopes of spreading chaos. In the mean time, border clashes are on the rise on the far side of the realm, a three week's journey there and another three back. Oh, and you'd better hurry, because winter is coming.
Looking at the world from this perspective will generally help you make time meaningful. As that time is invested in various story-based activities, I think you'll find the story growing in interesting ways.
And yes, it's definitely OK to hit Fast Forward sometimes, and just gloss over that week long trip. Or the boring winter spent
leveling up training at Castle Safekeep.