Use Disadvantage and Skill Checks
There's no need to house-rule this. DnD 5e is already set up to deal with these types of circumstances. The DMG suggests that you impose disadvantage when you feel it is appropriate and to make skill checks when there's a chance for failure. So, just like a player would need a skill check to do a backflip mid-combat, you can make your players make a skill check with their magic casting ability to pin point a particularly challenging fireball.
Disadvantage on attack rolls
5e is balanced primarily upon Advantage and Disadvantage. It's effectively a +/- 4 on your rolls statistically, but with all the excitement of rolling two dice. It also does not stack and Disadvantage and Advantage cancel eachother out. This means that no shot is too difficult for the player. At worst, they're rolling at disadvantage and at best, they're rolling normally (provided they acquire advantage). Rather than modifying AC, where such circumstances can make a shot nearly impossible (such as a +4 to AC at disadvantage), you only impose disadvantage. It's simple, it's quick, it's not too difficult. The drawback of this system is, as you've probably noted, it's a binary system of imposing difficulty. You either have disadvantage on this attack, or you don't. So you're losing out of the variability of your 4-point cover system (zero, half, 3/4 full). However, I think the simplicity and ease of use of disadvantage is worth it. It's precisely what the system was designed for.
If there's no attack roll, use a skill check if it's particularly difficult. The rule of thumb for skill checks is that they should only be used if there's a chance for failure, especially if that failure is interesting. For instance, you don't need a skill check to break down a door if you've got 8 hours to do it. There's just no need to keep rolling, it's assumed that eventually you'll get it down. Similarly, you don't normally do a magic skill check for fireball and the like because you're already really good at doing it and it's second nature to you. Placing the fireball is easy-peasy under normal circumstances.
But when you have a lot of obstacles to thread the needle through, it might behoove you to require a skill check, because
- Combat is stressful
- Failure is interesting
- It's not easy.
I've seen this used plenty of times and it makes sense. Have your player make a "spell" skill check. Just have them take their spellcasting modifier and add proficiency, like if they actually had proficiency in a skill called "spells". But they don't, so we're making it up. This is more or less a house-rule because it's making up a skill, but beyond that it's perfectly vanilla. The DC they want to hit is entirely up to you. It shouldn't be terribly difficult, but the option for failure should be present.
What to do on a failure? If you want to get really in depth, this would make a great question by itself. But, here are a few ways I've seen it done:
- You can "scatter" the impact zone using a d8 if the spell picks a point in space
- You can pick another target the spell hit instead using the same scatter rules if you're targeting a creature
- Just get creative with it if it goes poorly. This is what you're there for!
You can also use this perceived randomness to help or hinder the party if combat is going too well or too poorly for your liking.
Worst case scenario, you can argue that a target cannot be seen and therefore a spell cannot be cast on it that requires line of sight.