You need a session zero!
Instead of running the first session when people get together, let them know you want to gather them to explain the rules, the setting and the buy-in.
You can use this to explain any homebrewing or quirks you have as a DM (no talking over each other, give heads-up if you can't come to a session, and oh by the way I like rule of cool and run a high lethality campaign!), lay down the basics of the setting and plot (so nobody makes a elf ranger in an all-urban campaign in a city that hunts elves, as an example), and most importantly throw down the buy-in.
What's the buy-in?
The buy-in is an all-powerful, magical tool. Players should always have ownership of their character, but buy-in is a way to veto toxic characters before they are created. Basically, you just explain some core assumptions and requirements of the characters. This is really useful on multiple levels. It can inform the general party alignment (you can be evil, but the group is going to accomplish good things), make any character with any background fit in the story (I don't care who you are or where you come from, but you need to care about [Insert first plot-hook here]).
Whatever else I include, every game, I throw in a blurb that mostly goes like this:
You can be anybody in the group. You can be a lazy, selfish lone-wolf that compulsively steals from everyone. You can be a violent, unhinged person with temperament issues. You can be the most annoying goody-two shoes to ever exist. But whatever you make, you need to make sure they are cooperative, helpful and stick with the team. They need to respect their allies with their actions and (most of) their words. If the character in question would normally not act like this, it is up to you to figure out why your team is the exception. I don't care what it is - curiosity, history, blackmail, a bribe or even something never addressed in-game. If you want to roleplay tension within the group, clear it with everyone before it starts, and have a rough plan to resolve it within the current session, or the one that follows. This is a co-op game, and we're all investing time to have fun as a group. If you or your character don't align with it, you need to adjust until there are no conflicts, or you need to find a different game.
After the buy-in, as a buffer you can open the room to questions or discussion about the campaign or anything you just said.
In my experience, this works well
I've been playing and running TRPGs for almost two decades, with dozens and dozens of campaigns, hundreds of people and more than a few handfuls of those I considered to be bad players. What sucks is it feels like the onus is on you to be the bad guy and worse yet, sometimes these bad players are good friends, or people who are struggling.
Session Zero is a sneaky and powerful way to give them a clear opportunity to fix their behavior or back out. If they do neither and continue to cause issues, you've given yourself all the ammunition you need to fire them off into the horizon while still being reasonable. With the added bonus that a good Session Zero can also smooth over all sorts of other potential issues before they ever happen.
The only other thing I would add is a recommendation to not stare down the bad player while explaining things you don't want them to do. Don't stare anyone down during that, nobody responds well to feeling targeted.
Hope this helps!
What if the campaign has already started?
RyanfaeScotland made an excellent point about how this doesn't address how to have a session zero with campaigns that have already started, for those who need one. So I'm adding onto this on the chance someone reading this needs that advice. I've had to insert session zero into multiple campaigns after they started, and it can still work. In fact, I think in long-term campaigns, session zero is a thing you should use every now and again, to keep the group on the same page, as the context and dynamics of the game change over time.
Time and time again, the easiest way to insert a session zero mid-campaign for me is by taking a break and skip a few sessions first. I suggest replacing them with a few cooperative board game nights, where there will subtly be virtually zero reasons to be uncooperative - sometimes TRPGs get so serious, people stop thinking of the sessions as a fun cooperative event. Combined with the time away, it's a great way to let them mentally reset, which will make it easier to suggest changes.
The process is largely the same, but instead of introducing rules, setting and buy-in, you are clarifying them. This also has the benefit of being a nice, natural point to discuss changing any character, not just problem ones. Some players might not be having fun with a level-up choice they made, or they aren't enjoying the class like they thought they would. It happens more than you think, there are often players who want to make changes but are afraid to try to bring something like that up mid-campaign. If you present this as an opportunity to adjust characters for free, players are often more accepting of you presenting new buy-in terms.
Players may still kick back against new clarifications or buy-in. This can be a challenge from player to player, but the most common argument I'm given is "It's now how my character would act". In those cases, remind them the character continuously exists as the player defines them, and that the game is supposed to be a cooperative, fun outlet. If the character doesn't contribute towards that goal, and they aren't willing to change them to achieve that goal, then they are welcome to make a new character instead.
If nothing else...
Sometimes it might come to a point where you'll need to refer to Opifex's answer, and simply resort to a more direct response. It's not my favorite option, but I think it's the right one when nothing else works. Give warnings, foster the game in a way that ignores the bad behavior, and if it comes down to it, boot them. It can be really difficult...but look at is as weighing one difficult conversation, versus dozens and dozens of hours of difficult sessions with that person. In those cases as a DM you have to take charge. Let the trouble player know that they need to find a group that is interested in playing the same game as them.
