You need not change anything about the default setting in order to have people "left out" of the benefits of clerical magic.
I don't think that clerical spellcasting is as easy to come by as you make it out to be. In other words: you can easily have your harsh and gritty world. In what follows I'll always lean toward the more-utopian interpretation of things, lean on the rainbows-and-unicorns end of numerical ranges, and I think you'll see there's still plenty of room for gritty/harsh living conditions. Obviously, then, if we dial back any of those happy-happy-joy-joy assumptions we can make things even harder for people.
You've told us you want a harsh, uncaring setting. Luckily, much of our history on this planet has been conducted in that setting so we've got lots of data to go on. I'll point you toward my favorite such tool--Medieval Demographics Made Easy--which helps us imagine D&D's pseudo-medieval implied setting using the tax rolls of 13th-C France.
This gives a baseline of one "clergy" per 40 population, and one "priest" per every 25-30 clergy. For your purposes I'm going to interpret "clergy" as an unleveled acolyte, and a "priest" as a leveled cleric.1 That means that we find a leveled cleric every 1000-1200 population.2
Villages (according to MDME) top out at about a thousand people, so let's assume we've got a 1000-person village with one L1 cleric serving as its priest. That's two first-level spell slots per day for a thousand-person village.
In the case of illnesses remember that we can only cure two people per day. I don't feel like running a full S-I-R model here,3 but we don't need too-extreme of infection and recovery rates to overwhelm the "bonus" recovery from spellcasting. And curing an illness requires a second-level spell slot, requiring a third-level cleric!
In the case of extreme injury, just remember we only need three people seriously injured to overwhelm our poor country cleric. I've never erected a barn or taken down a tall tree, but I can easily imagine three simultaneous injuries. If a cleric can heal two but has to wait until the next day to heal the splinted broken ankle, there's your limper.
Accidental death. Let's talk size: sticking with medieval France we get a population density in arable lands of about 100 ppl/mi\$^2\$. So this village and its lands occupy about 10 square miles, or about a 3-mile (diameter) hex. Recall that without help you may only have 21 seconds of life left once you hit 0hp. The odds are not good that this accident happened within 20 seconds of our cleric =(
Cities (again, using MDME's classifications) range around 10,000 people.4 Using the same numbers as above we see that a 10K-person city should enjoy only ten leveled clerics.
But notice that the underlying numbers don't really change all that much. Travel times may go down, and with more casters we'll have some of higher levels (and more slots). But there are just so many people...
Again being generous, let's assume one cleric of each of levels one through ten, and that they use fully half their slots on daily petitioners. That's 45 spells to cast for ten thousand people, or 93 spell-levels for hoi polloi. All you need is more than one percent of the population to "need" something on a given day and you'll overwhelm the daily petitioning system. 2% would overwhelm the entire clerical-magic capacity of the city. One disease, a fire, even just a day when one of the high-level clerics is tied up in bureaucratic argle-bargle will leave some unfulfilled.
Turning the dials
Let me briefly summarize the assumptions made, and how changing them will change things:
"Clergy" are not casters, only "priests" are. They are as abundant in D&D-verse as in medieval France. Obviously, in a world where the gods walk the earth there might be more priests. But keep in mind (a) there's also the lure of arcane casting; (b) clerics are also needed to deal with the undead hordes/curses/magically-fueled religious wars inherent in such a world; and (c) somehow the D&D-verse hasn't figured out any better agricultural technology than the moldboard, so we still need 90% of people to be involved in food production.
Farmland is really rich so the region/kingdom/world is pretty densely-populated. Drop population density down to 30 ppl/mi\$^2\$ (like 13th-C. England) and one's proximity to a cleric drops commensurately.
City-based clerics spend half their spellpower helping "commoners". This one I see as a fun dial to twist. If I know anything about bureaucracy at least one or two of them definitely aren't getting their hands dirty. Perhaps some orders are martially focused and so are embedded with a standing army and none of their clerical spells go to (directly, immediately) help common folk. Perhaps some order devotes 100% of its divine power to such works. Perhaps some order has grown world-spanningly-huge and spends much of its time maintaining its own bureaucracy. These are huge cultural differences that can really change how the world feels.
The cleric in a village is a L1 caster. But one can imagine that a village where a L15 cleric has retired would be a little green patch of paradise: he attends all childbirths, so mother and infant mortalities aren't a thing. Droughts/blights? Create food and water'll sustain people and seed crops well enough to get through. Village festivals (to the cleric's god, obviously) come with a Hero's feast.
Bringing it back to reality:
The situation you describe isn't fundamentally different from our own world. There are people with physical troubles, and there are some with the power to alleviate those troubles. Why, then, do we see people in difficult circumstances? Proximity, access, cost, &c all play their roles. The same is true in the D&D-verse.
In some places charities step in. In some places social structures embody greater sympathy/antipathy toward the problem. But spell slots are a limited resource and so will evolve into their own economy, just like anything else.
1 - This, effectively, is exactly what @timster argues for in his answer.
2 - If it seems I've queered the deal from the outset by declaring "priests" to be leveled clerics rather than "clergy" (and thus dialing back by a factor of 25 the number of available clerics), I urge you to consider the alternative: 2.5% of people are leveled clerics. Given twelve classes and assuming clerics are one-twelfth of leveled characters we see 30% of the population is leveled.
Sticking with pseudo-medievalism we've got to have ~90% of the population involved with agriculture, though. So not only could you have every non-farmer a leveled NPC, one out of every five (roughly) farmers would still be, too! That's not really the setting implied by most published material. (But it's kinda a cool one, methinks. Every time you stop for dinner you've got to listen to the barkeep's old campaign stories, then the stablehand's old campaign stories, then the lamplighter's old campaign stories....)
3 - okay, that's a lie. I really want to run an SIR model incorporating a clerical healer and I did. Assuming the cleric (a) doesn't get infected ever, (b) can get to and heal two people per day, (c) healed people now have immunity and can't re-infect, and (d) the same epidemiological parameters as the SARS outbreak in Hong Kong in 2003\$^5\$, we get a peak infected population of ~15% the total population; 150 people have this disease on the outbreak's worst day. The cleric, healing two a day, just doesn't do very much for those people; there's plenty of room for some disfiguring illness to scar people. (The cleric does have a huge effect on ramping down the "long tail" at the back end of the outbreak.)
Admittedly, the underlying SIR parameters may be badly constructed for this setting: people are less-densely packed and don't travel nearly as much. But then again, hygiene and sanitation are much worse and the population may be starting at a less-healthy baseline. In short, I'm not a medieval epidemiologist, just an amateur mathematician with a copy of MATLAB.
4 - This may seem small to us--it's the size of the suburb I grew up in, for instance--but "a typical large kingdom will only have a few cities in this population range." (MDME)
5 - Razum, Beecher, Kapuan, Junghauss. "SARS, Lay Epidemiology, and Fear." The Lancet, May 2, 2003 + analysis of these data in one of my modeling classes in grad school.