I'm running 5e on a grid with minis. The one thing that I hate is when
a sorcerer or wizards casts a fireball that perfectly hits enemies in
the aoe. The player counts the squares on the grid to determine
exactly where the fireball can hit and he knows that the perfect way
for the fireball aoe to take effect. It will hit enemies but amazingly
the blast stops just in front of an allies face. A player who was
attacked by 2 melee enemies in my game casted a fireball behind the
enemies so that the enemies were hit in the explosion but he wasn't.
In my opinion, this loses verisimilitude.
There are few meta-issues here that I think are at the core of the issue...
"Verisimilitude" - The game is about fighting fire breathing dragons; thwarting the machinations of evil gods; defeating vampires, liches, and other undead; riding a Unicorn or a Griffon; and, yes, casting and manipulating magic! All of those things require a fair bit of imagination to conjure, let alone play in any meaningful and balanced way. The game allows players to throw fiery magic balls of Troll-death and its fun!
Sorry, but verisimilitude is out the door, go play Twilight 2000 if you want realistic combat.
Keep it going...
That said, I do agree there is a problem with players planning the perfect circle AoE to only hit the enemies. The issue though isn't the spells, it is that play virtually stops while the player (usually players) figures out the perfect spot to land the spell.
I see Wizards as having a clear idea of what their magic can and cannot do. They will know their limits as clearly as an Cleric healing the wounded, or a Thief picking a lock, or a Fighter sizing up an enemy. But unlike the Cleric, Thief, or Fighter - the Wizard can pin-point their magic... and that's fine. Although unlike the Cleric, Thief, Fighter, or any other class, a wizard without spells is about as good as a pitchfork wielding commoner. Let them have their moment.
However, I think if you gave the player about as much time to pin-point their attack as you give the Fighter to roll their attack dice, I have a feeling you'll see a lot more PCs getting roasted, or at the least more enemies surviving. Just because the Wizard can pin-point an attack does not mean they get more time to think about the best place to pin-point. With that in mind the PC might fudge a bit further away just to keep PCs out of the way - but that means engaged enemies might be missed too!
Yes, they can pin-point. But if you keep the play going... they won't have time to.
Creating... crafting... house rules are something we all do. And a lot of those house rules can make a bad situation worse. I've learned both professionally (designing computer games) and in my own table-top games that a level of care must be taken when Crafting Rules, especially those designed to 'fix' other rules.
I've seen several suggestion here, as well as elsewhere, that are to impose new requirements or restrictions: randomize the pin-point, adding a skill check or attack roll, applying modifiers based on stats, etc.
- Random Pin-Point: In these rules, the desire is to add a random factor into Magic that the other classes have to endure. The main problem in doing so it can nerf the Wizrd - remember, the Wizard has a finite number of attacks while the other classes do not endure this restriction! The power of the Wizard class is the inherent ability to pin-point an attack, taking this away is actually a severe limitation.
- Skill Checks or Attack Rolls: These can be hit or miss, usually miss, as a Player generally invests in Int or similar skills and forgoes combat prowess so as to maximize their magic (& pin-pointedness...) Applying these rules at character creation can be fair in so long as the skills/attacks are something the Player could choose normally and don't detract from the Wizard's primary focus -- throwing Magic. Applying these rules after character creation only slams the Player for pre-game and in-game choices already made, causing a new problem... an unhappy Player!
- Stat modifiers: Unless the stat in question is Intelligence, don't bother - you end up with the same results as Skill/Attack above... an unhappy Player.
After learning the hard way, I steer clear of house rules that modify X rule with Y rule since doing so invariably causes new problems (usually with unhappy players because of the new restrictions to any desired type of play).
I hate is when a sorcerer or wizards casts a fireball that perfectly hits enemies
Sorry, but that's what magic users do, and you as GM should let them. If you hate it so much to create limitations, the problem isn't the rules, or the players knowledge of said rules...
Don't create rules that restrict, limit, or hamper a Players' desire to play the way they want to play!
Crafting Encounters: Dming Advice
How you craft your encounters is by far the best solution to Players use of AoE.
Yes, it isn't a rule per se and yes, it doesn't address players pin-pointing their attack (see above). But it is the best thing you as GM can do to reasonably change how Players use pin-point AoE In a way, this is the best rule of all -- you get to use your imagination in crafting interesting encounters. A great deal of the the issue of players using AoE spells can be mitigated when crafting the encounters in the first place.
Anyways, what's good for the goose is good for the gander! Why aren't there any enemy Wizards lobbing balls of fiery doom exactly where only the PC's get hit?
I strongly urge you to NOT apply any rules to address the issue. Just keep things going and keep the combat interesting and yeah, occasionally the Players will fire off a Fireball that only roasts the enemies. That is because doing so is a lot of fun - and your primary job as the GM is to ensure that that is what the players are having... FUN!