I'm a new DM, with a group of new players. I am the most experienced of the group, but mostly from watching Critical Role. There is a player in my group who multi-classed two homebrew classes; Witch HunterWitch Hunter and Dragon Slayer. Adding insult to injury he didn't prepare at all, and what he did know was overpowered stuff. So we said, look, find a different, simpler class and now he's a half-devil (specifically not a tiefling however) hellblazer.
It's mostly a difficulty of familiarity. When he played the dragon slayer class, he reeled off spells that were overpowered and because they are technically in the list I suppose he could very well use them but I think the difficulty there is obvious. Essentially he is asking a new DM to work without any real rule guides; he wants to be a half-devil but there's no race information for me or him to look over, not to mention how difficult it's going to be for him in cities and stuff.
What should I do to control this situation before I kill either him or myself?