At what time would it be "ok" for me to ask these disruptive players to step down from the group?
Well, the simplest answer is, "Whenever you've decided you've had enough."
The longer and probably more accurate answer is, "Whenever you've had enough and have enough support from the rest of the players that you won't shatter the group by dis-inviting them." Because that is always the concern in my mind: Gaming groups are often groups of people who are friends already, and dis-inviting someone from a game can often have social blowback both in the game (good players feeling like they have to take sides and support the other players) and outside the game (hurt feelings, splintering real world friendships, etc.)
In this case, that doesn't seem like much of a problem.
Is there any way to deal with them that isn't just asking them to leave?
Nothing that is guaranteed to be successful. You cannot force people to take something seriously (in attendance or in tone) if they do not.
What you can do is sit them down and explain to them that running a game is hard work, and that continual recaps due to absences make your life harder, as do constantly trying to keep the tone of the game where you want it. You can also point out that, as they obviously know due to their poor attendance, leisure time can be a scarce commodity these days, and their constant recaps and wandering off-tone is actually reducing the amount of fun that the other players have. (Make sure they are actually on board with that statement, though.)
In other words, guilt trip and implicitly threaten to kick them out-- the first time-- without actually showing them the door.
You might also ask if there are reasonable accommodations that can be made to get better attendance, although in a group of six, coming up with a scheduling change is probably going to be very difficult.
If your other players are game, you can also let them start handling the characters, but with the implicit understanding that... while you're not actually trying to murder them, per sekill the characters, the handlers aren't expected to put the interests of the absent players' PCs above their own, or even equal to their own. If they end up being damage soakers, who get less than a full share of experience and loot, and use precious resources, well... so be it. (I am not above murdering a PC when annoyed, mind you-- I just don't implicate the other players in itNote that this may go over poorly.)
When push comes to shove, though... be ready to axe them. Don't make a bluff if you don't want someone to call it.