I have someone like this in my group. He must always play something outside of the rules, and refuses to play a majority of the regular races and classes. Now, I have found that putting him through danger or discriminating against him usually ended with him winning and thinking he is unstoppable. So, I have found 2 options that work.
Deny his homebrew creations. This is the less preferable option, as it usually leads to resentment and anger. However, it does solidify your position as the one in charge. Before you just shoot it down, read up on the class beforehand. Think about how it could combine with other classes, and what its strengths and weaknesses are. If you can't see any weaknesses or notice strengths beyond that of other classes, veto it. Also, think about if it fits in your world or not. If you are playing a low magic medieval campaign, don't let him play a gun owning wizard. Remember, heroes may be special, but that doesn't mean they get to break your rules and your world.
Have his differences stand out. My homebrew friend once played the "Drunken Brawler". His character began outpacing all the others in almost everything. He broke through walls, broke fights and generally caused resentment. Eventually the party split in game. Instead of just making the world hard for him (which is a good idea) and have him hate you, have him be different than the party. This works differently based on party. Some parties might silently resent the person, some might confront him and occasionally it will do nothing. This method discourages the use of overpowered, unbalanced characters through social pressure and/or awkwardness role playing in game.
Now, these methods tend to alienate the homebrew character. Just remember, unless he is definitely breaking the game and upsetting the balance, let him be creative. Homebrew is fine with DM approval and balancing issues solved, as it lets heroes be unique and creative. And you don't have to allow homebrew if you feel it leads to too many problems.