Based on Rob Schwalb's article Stuck in the Middle:
How do you, as a player or DM, deal with choosing or forbidding broken combos in your character, group, and system?
This is only a partial answer, but sometimes, just sometimes, I do nothing if the effectiveness is logical.
For example, suppose players realize that if they build huge stone walls in a rectangle with a single gate with boiling oil over the top, and place archers on top of the walls, they're extraordinarily well-defended from almost all ground-based threats, even those that would wipe them out in a flash if they were in some other structure like a house. Instead of coming up with massive hordes of stone-chewing giant beetles, or critical-fault tables for stone walls, I would probably say, "Yes, actually, that's a castle, and was very effective throughout the middle ages. Good show! But you have to come outside to do many interesting things."
If it's a single power that's broken, then I'm more inclined to fix it. But if it's a combination--and the player is not being a jerk about it, and each of the pieces makes sense--I usually try hard to let it be and shift the nature of encounters to reduce the magnitude of the problem.
For example, if a necromancer has good life-drain powers, but they don't drain fast enough to really keep the necromancer alive against a tough opponent, and a mesmerizer has good dazzle and slow powers, but can't take enough hits to survive a tough opponent, you might consider a necromancer/mesmerizer broken, as they can slow the opponent enough so that life-draining allows the n/m to toe-to-toe with quite powerful opponents (one on one) forever.
Personally, I'd react to this by making sure that most powerful opponents have enough friends, not by saying that necromancers can't learn mesmerizing, or by saying that life drain can't work on slowed opponents, or anything like that.
Generally I try to define, with my players, what "broken" really means. If everyone in the party goes with a fairly optimized build my PCs may be more powerful than average, but it isn't hard to create a positive roleplaying experience by simply writing stories that put them up against more powerful antagonists.
Usually though, there are a few abilities that seem completely out of whack to most members of the group. These sorts of powers have a few common characteristics:
Best case is I know the system well enough that I can resolve some of these problems before we even start. I just let the players know when making characters that some abilities are not allowed or that I've reduced their effectiveness.
For example: The Exalted Twilight Caste power decreases damage taken at high anima levels. I've found this to be unbalacing in the past and so when folks are making characters I just let them know I've revised the rules.
Occasionally a broken combo arises in game and catches me by surprise. Usually though, this is pretty obvious to the entire group. When the session is winding down I've had a lot of success bringing up broken combo as a discussion point and then asking the group what they think about it, particularly in the framework of the points enumerated above. Most players want a balanced game and usually by group consensus we can tweak the rules so that it is less effective, or has new restrictions.
This answer basically boils down to "Talk to your players", but you'll want to frame the discussion in a way that doesn't make the player with the combo feel unfairly targeted.
I avoid the issue. I invite players who play characters rather than builds. I run stories designed to entertain characters rather than builds. If somebody shows up with a combat monster, they're probably going to be bored in my games. I also make sure to tell the players this upfront when I invite them to the game. It helps them get into the mindset of playing a person instead of a set of statistics.
Play a game that is not as mechanics dependent. There's a reason that discussions about "broken" only really appear around D&D 3e/4e, Rifts, Exalted, and other games that have published enough rules content to choke a horse with. There are literally hundreds of RPGs where the phrase "that is a broken combo!" have never been uttered about.
Simply exert your authority as GM. Say "that power makes encounter design and/or being another player at the table a pain in the ass, so I'm going to have to disallow it."
Find an in-game world response to the problem. If grappling is the ultimate anti-wizard tactic in your game system, aren't all wizards going to know that? And aren't magic items or other countermeasures going to be highly prioritized and commonplace? Duh. If it's not something general but specific to a character, it'll work more but any well informed foe of your PCs will prioritize counteracting it.
Example: In my current Pathfinder game, our druid got a big ol' snake companion. He convinced me to let it take some feat it didn't qualify for, forget exactly which one, that made it an even worse grapple monster than usual. The thing slithers up, bites, autoconstricts, does 30 points of damage in a round easy. They're only fifth level so that's one round kills for a good percentage of their foes.
I considered taking the feat away, but that might or might not have actually solved the problem in this case; instead I decided to just make sure and continue (I had already begun) to enforce a certain sense of realism on his animal companion - it's not a killbot that will just leap from foe to foe every round and crush them, regardless of how tasty a thing it has in its coils already, whether the foe is a demon or on fire or spiky or otherwise unpleasant. It'll reliably go for someone at start, but then requires Animal Handling checks as standard actions from the druid to do anything an Int 1 snake wouldn't normally be doing. It tends to attack someone hitting it and ignore anyone not hitting it, especially if it has something in its coils already, and thinks very dimly of attacking anything that doesn't fit its definition of readily edible.
It has worked out well. He still loves his snake, and it is very effective in certain situations, but also ends up being "left behind" in some cases or not participating effectively in some combats. So not broken, but he also doesn't regret his choice. Voila, balance.
There are basically two kinds of broken.
The easiest to correct is the kind that is always broken. Whether a combo or just a strong move it will be something that is easy to notice and so a quick talk with the player and you should be able to either drop it or house rule it into being fixed. If the player does not want to just let it be fixed you may have to be a little heavy handed but that's part of running a game. If the whole party is using it then warn them that you can use it just as easily as them if they don't want it fixed. Generally the thought of their craftily created combo used against themselves will make them see the light and if not the actual use of it should.
The other type is the situational type of broken. This is harder to correct but easy to fix. What do I mean? Correction is where you actually change the rules and requires the players in some form while a fix is just something you can do without going through the players. An example would be an ability that knocks out a large number of humanoids. All that requires is that you include more non-humanoids and the problem is solved, this would mean if you happened to roll bandits for a random encounter you include some attack dogs in the mix. Now don't go and over correct with this though. If you never again include any humanoids the player will complain about it and rightfully so, you just ruined an aspect of their character. It gets harder to fix the more broad the ability is but after a point its not in this category anyway. If a character is amazing at fighting throw in a number of role-play type encounters or a bunch of low powered enemies so that while the guy may be able to power through them the rest of the group still gets some action.
In the end though what matters the most is that everyone has fun. What you see as broken may be seen as quite amusing by everyone else or it may be in reaction to you making the game so hard that they needed to look for such combos and moves. Something may seem to be a fact to you could just be your opinion so the input of everyone is import, after-all its not just you playing the game.