While playing a game with a previous group, in the middle of a combat, I had an enemy caster hidden from general combat, but still casting different spells like heal and other non-damage spells to support the enemies. My PCs never thought to look for him and were becoming frustrated when I would seemingly randomly add health or other effects to the ones they were engaged with.
To make matters worse, I rolled a dice to randomly pick a party member and just told that character to make a will saving throw without telling him why. When he failed, I handed him a note that told him the enemy I was using had just crazed him, and on his next turn before attacking he had to roll a d6: 1-4 was party member he had to attack, 5 was he attack an enemy, and 6 was damage self in some fashion. He rolled a 2 and attacked the corresponding PC. He ended up dealing a critical and killing that PC. At that point anarchy broke out at my table with ensuing fights and eventually two people getting up and just leaving.
The caster wasn't impossible to see; the PCs were just more concerned with beating things up and I tried to indicate he was there. At one point, I did everything but put a giant flashing sign saying "there's someone in that thicket". Three different times, the guy's raven familiar flew from that location, touched an enemy, and returned to thicket. At the end, while explaining to the players, I asked them if they never thought that was suspicious. They all told me they thought it was just a bird and wanted to know how they were supposed to know it wasn't just a regular raven. There was even a sorcerer in the party who had a familiar, who never said anything during the encounter; he just kept spam-casting.
I didn't even intend on this situation to be a combat encounter. The PCs were supposed to be looking for the enemy that was in hiding to obtain some info. Instead, they got to a camp and never stopped to figure out if the people were hostile or not: they just got murder happy. They all knew the spellcaster 1) should have been there 2) was a high powered individual 3) was a recluse who was opposed to conflict.
My question is: was I wrong as DM to force the player to attack other characters? Is that within my power?