I have a bit of a problem, particularly relating to a custom monster I created. D&D has a lot of effects and abilities that change a character's behavior, ranging from a simple intimidate check to a spell in the Book of Vile Darkness called "Mind Rape" that lets the caster entirely rewrite their target's consciousness. This monster that I created, Mind Rape is exactly the kind of ability she would use, it's entirely consistent with her nature and concept. The problem is that I'm afraid if I so much as make an intimidate check against a player, my friends are not only going to stop being my friends, they're going to flat-out crucify me. How, as a DM, can I run a game that includes fear effects, confusion effects, charms, and compulsions without my friends hating me for controlling them?
Fundamentally, this is about learning player expectations and limits. You said it yourself, here:
The problem is that I'm afraid if I so much as make an intimidate check against a player, my friends are not only going to stop being my friends, they're going to flat-out crucify me. How, as a DM, can I run a game that includes fear effects, confusion effects, charms, and compulsions without my friends hating me for controlling them?
There's only one real way to answer that.
What type of game do they want to play in? Is it one where effects like mind rape are in play, or one without any form of mind affecting spells, or something in between? You're afraid to use it because you don't know, and the only way to get past that fear is to talk to them about it.
Maybe they really do hate this type of thing. If that's the case, you'll want to seriously consider avoiding using such an effect (making your players uncomfortable with something like this won't help your game). Maybe they're perfectly okay with it. It's not something we can answer for you, because we don't know them. With that kind of effect, there are a lot of different player reactions.
It could be as simple as sitting down as a group, explaining what mind rape does, then asking if they'd be okay with that effect being used in game. If they tell you they have out of character problems with it, then that answers your question. If they say it's fine, that likely also answers your question, so long as they're not just saying that to conform. I'd encourage anyone with an issue to speak up privately if they don't feel comfortable doing it in the group setting. You can also ask them one on one via email or messaging if you have players shy to speak up in front of the group.
There are tools out there to help with that conversation, and this question's answers can help you with that. I'd suggest you take a look there for tips.
With my players, I would never use mind rape. They invest a lot of time and effort into their characters, and get attached to them. If they die in heroic combat? Okay. If the player does something foolish and that gets them killed? They tend to be able to accept that in time. (One of my players wants me to try to kill him, because he wants it to be a real challenge so the triumph means that much more.)
But what mind rape does is more insidious, and it alters their carefully created character in ways that can very easily make it not-fun to play for them. I won't deliberately make the game not fun for my players, so I just won't use it.
I will use spells like charm, as that's a temporary effect that can be fun to roleplay, and my players readily accept that. They also have been known to use it.
I've played in other games with players who wouldn't have a problem with it, though. It really depends on the player.
That spell is particularly bad for the players, as it allows the GM to essentially rewrite the PC and force the player to play what is, to him, another character entirely. You could, however, use it in a somewhat less damaging way...
In movies and literature, amnesia is a common trope. Between adventures, the PCs could be enjoying some downtime and go meet an old friend, who is very excited to see them again, as it's been over a year that they visited him.
"Didn't we see you like... last month?"
"I wish it were so, my friends, but the last time I saw you was when you defeated the necromancer Tur'azad, over 12 moons ago!"
"That was definitely last month..."
Now your PCs will believe that they have somehow traveled through time or something, but you could lead them on with some clues until they realize that they, in fact, have really aged one year, of which they have absolutely no recollection.
Unbeknownst to them, they were the subject of a Mind Rape spell (or other such strong mind controlling effect), and the bad guy used them as tools for a year or so. He then discarded them, erasing (or worse, rewriting) their memories of what happened during that time. When the PCs start investigating, they realize they have a very bad reputation in many places...
While not perfect, this setup allows you to enjoy some of the benefits of controlling the PCs through mind-affecting powers (they worked for the bad guy, after all), yet does not rob them of their freedom to control their character as they see fit. Indeed, since these acts happened "off screen", the players never had to role-play their characters in a way they didn't enjoy. And now they have a very good incentive to go after your bad guy while taking a lot of precautions, as they can never be certain that they truly are free of his influence.
Keep it simple, keep it light, keep it fun
You want to include mind-altering effects in your game? That's great, they can be a lot of fun for players and GM alike. But to use them and keep it fun for everyone can be quite tricky.
Keep it simple
Mind control like "You feel a strong compulsion to steal everything that isn't nailed down" or "You have an overwhelming fear of sheep" is easy for a player to roleplay. On the other hand, if your players wanted a game where you explore the inner workings of characters like Jason Bourne (not the best example, but hopefully you see my point), you probably wouldn't be asking this question. Of course, if your players do want this kind of game, then go for it! But make sure they do, and consider changing to a system that supports this a bit better.
Something simple is easier for the player, which means they're more likely to make the most of it and really have fun with it.
Keep it light
No implanting memories of murdering your own parents. This is the sort of thing that you definitely need to consult with your players about before including in your game. Any elements that are really dark or horrifying need to get group approval before you even consider including them.
Likewise, anything that represents a real psychological issue is probably a bad idea. There are too many ways this sort of thing can go horribly wrong and spill over into the real world, and you never know whether one of your players has had serious problems in the past, or could even be having them now.
Keep it fun
Your players will forgive you almost anything if they're all enjoying themselves. Note that this means never taking direct control of a player's character. If you force them to betray and attack their party, they'll probably really enjoy that. But if you push them aside, take over their character and leave them sitting there with nothing to do, they're not going to be having fun.
I recently DM-ed a session where my players fought a group of yuan-ti. I kept casting Suggestion, and the players who it worked on had a ball pushing the limits of the orders they were given, seeing how they could work within the words used, and just generally playing the game to enjoy themselves. When ordered to attack his own party, one player immediately attacked himself until he was unconscious, since he was the closest member of his party. He removed himself as a threat and everyone thought it was hilarious. But if I'd said "No, you have to attack the others, you know what I meant", it would've sucked. He'd have become just another enemy.
Keep it simple and keep it light are really just methods towards the goal. The point of a game is to have fun, so as long as everyone around the table is enjoying themselves, you're doing it right. Most players won't mind their character being controlled in-game, as long as they still control their character.
On Mind Rape specifically
It's probably not something I'd ever use on a player without their permission. But try asking them, they might be perfectly willing to give that permission. For example, if one player is sick of roleplaying a goody two-shoes paladin, he might be really glad for the opportunity to rewrite his character into something more interesting. Cooperation with your players will make it possible to do this sort of thing without upsetting anyone.
Personally speaking, I would (and do) play with the players, not against them. If your players are playing against you in order to "win" the game, these things (having your characters "influenced" by their counterparts, either subtly or by force) do not really work.
But if you are playing together on the story, you can make the players part of the story. Take the target of the Mind Rape aside beforehand (e.g. during a coffee / snack / pizza break), give him a general idea of what is going to happen, reassure him that you're not out to kill off his character (important -- you are not playing against your players either!), make sure he has understood the plot idea and is OK with it, then have him play out the personality change of his character in cooperation with you.
This requires trust, in both directions, but can be great fun for all involved. The targetted player because he's getting to play "special" for some time, and might actually come up with cunning ideas of his own, on behalf of his new persona. He has to trust you to not carelessly let him killed, because he cannot take action to that end anymore. In exchange, he's helping you creating a great situation for the group.
The rest of the group might be taken completely by surprise, and have a great time figuring out what happened to their trusted teammate, and how to get him back to normal.
If you haven't played this way before, but would like to give it a try, this might require a "meta" session with your players first, to get a former "competitive" group into a "story" group.
It can be done, and it can have great results.
That being said, I want to explicitly point to the Keep it light section of Miniman's answer.
Post Scriptum: I am not familiar with D&D. From reading other answers, I came to realize that Mind Rape is a permanent, non-reversable effect. I agree with Tridus on this one: Such a spell should be treated the same way as an instant-death spell. It's one thing to tempararily switch a player's persona, it's a different thing to effectively end the existence of what made his / her character special.
While it still can be done, personally I would advise against it, at least as a plot mechanism. Treat it as the death spell it is.
I don't play D&D, but at heart this isn't a D&D question so much as a human "real life" one (as several people pointed out). It's very similar to a ton of real-life situations with the theme: "I want to tell my friends what I think/feel, but if I do, will they just get angry, or stop being my friends".
Other people have discussed how to bring this within D&D very neatly. But the social issue it reflects is probably worth a comment.
You secretly worry they may not like the course you impose on the game, and it may raise strong feelings. So here's the thing. Are you DM'ing to give your friends a chance for a game they will enjoy, styled to their preference? Or are they playing because it gives you a chance to enjoy control as a clever and powerful decision maker and provider of surprises? (Okay, could be both but people often assume the former much more). Don't most of us dislike when someone makes too many many assumptions about what we "will" or "won't" like, or "must" or "mustn't" really be wanting, and goes off without checking?
See what I mean?
It's incredibly common to think and act on the belief that one is trying to please others, when in fact really, one wants others to fit in with what would please oneself. Then one will (in fantasy anyhow) get all the feel-goods and plaudits for it... only perhaps that's not what they wanted. Perhaps that's not the D&D they assumed they'd get at all. You want a game with dark mind-rape spells where you get to dominate their play in this way as well... did you check if that was the game they wanted, or had in mind? Did you ask if any of them might have discomfort about the style? or did you assume they would fit in with your whims, even though they could have strong reactions, without feeling a need to check?
It's a mistake we all (most of us) make now and then. If you happen to have friends who are LGBTQ, you'll find that as a community, they're very aware of the risks of assuming about others preferences, and the advantages of asking people up front. For D&D the setting is different but the issue is very much the same: "I'm thinking of X, would you be okay with that", or "I'm tempted to run it a bit darker, with these or those features, do you have any limits you want me to set, or can I try it and see where it goes?" Then you are inviting them to see if they like your thoughts without apprehension.
The same respect will help you away from gaming, in every situation where it's worth a reminder that people aren't just reflections of each other, and that others might not think as you do, about one's own wonderful ideas, or often they may have different thoughts of their own. It'll be a good guide to have that approach of respect and awareness in work, social, relationsip, and other parts of life.
It may even help you get your dark game without fear of losing friends, at best it might let them enhance it by working with them, and - at worst, it'll let you safely discover if your friends really don't want to play it after all :)
A lot of this will depend on the players. Some really do hate this kind of thing.
But the way the DM lets a character be played when they are controlled can have a big impact on how much fun the player has when it happens.
If the DM just takes it over and runs it then the player is left with nothing to do except feel pissed off. They get to sit out of the game and watch their character do things they as a player don't want it to do.
But if the DM gives the player a briefing with some instructions on it and lets the player continue to play the character with the new personality, then it can be more interesting. Some players won't do a very good job of playing the new personality, but some will relish the role playing opportunity.