The Assumptions Behind Challenge Rating
I don't have my 3.x DMG handy for a page reference, but somewhere in there it states that D&D 3.x operates on the assumption that players have multiple encounters in a day.
Very few encounters with a challenge rating in the same ballpark as the average party are going to be likely to kill the party on their own. What makes an encounter "hard" is that it burns a large amount of resources... Magnifying all future encounters.
Consider this: A single human with class levels has a CR roughly equal to their level. This means that a level 4 fighter is an "appropriate challenge" for your entire level 4 group (at APL + 0).
Is one fighter going to have a chance at stopping another fighter, plus a rogue, cleric, and wizard? No. But he will do some damage before he goes down. He will force the cleric and wizard to use spells, or the fighter and rogue to use consumables.
The same applies to Pathfinder, except that a single fighter is CR3 (APL-1). For the sake of the argument, consider a level 4 fighter plus a level 4 cleric (total CR5, or APL+1).
The "standard formula" that D&D's CR system is balanced around, is a number of small (at-, below-, or barely above-level) encounters, followed by some number of larger, more challenging fights. Hit points, spell slots, consumables, etc. all are built to support this slow war of attrition.
Challenge Rating in Your Campaign
Even in the wilderness, you can still pack in multiple encounters around a given combat, as long as the players have purpose. Defeating or circumventing guards, wandering creatures, and other hazards can be stretched over several encounters.
If the players are backing out to rest after each encounter, remind them how much time is passing and adjust the world accordingly. Did they spend a full week working through a day or two worth of encounters by resting until the next morning after each one? Well, by that time the big baddie should be dug in, or long-since escaped.
Alternatives to the "Standard Formula"
There are certainly a number of ways to work around these assumptions. Using superior terrain and tactics, as Chaosys suggests, is one way to do so. Just remember that in doing that you're essentially creating an encounter of a much higher CR than what's printed in the book... I'd be pretty irked if I went through that hell, and only got rewarded for a handful of 1/4 CR opponents! :)
Another option is to simply inflate CRs... However, when doing this you should focus on including many "appropriately" CRed creatures over a few creatures with inherently high CR. High-CR creatures are more likely to become unhittable/unkillable/one-shot/otherwise frustrate players.