I had this issue with a large D&D group back when I was living in an apartment in Memphis, same kind of setup you had. There was a core of folks looking for more consistent, "serious" roleplay and there were the folks who, either out of interest or out of conflicts, couldn't participate much. And one guy who was a goon.
Eleven people, by the way, is a crazy insane untenable group size even if they were all present and on time and super on task every single game session. So you have several layers of issues to fix:
- Group just plain too big
- Casual vs committed members
- Specifically disruptive member
What I did to solve #1 and #2 at the same time was, after talking with all my players to figure out what they could do and what they wanted, was to "split the group." I ran a Sunday game for the super committed folks. Then I had a Wednesday night game for the casuals. The Sunday game had group expectations, including "you need to make 3/4 weeks or this isn't for you." I didn't drive this, the players who wanted that kind of RP experience did. The Wednesday game was casual, board games if not enough people showed up, more about getting dinner and talking. I didn't even run it all the time, we rotated GMs (after some prompting by me, because running two games even if one is casual is a lot of work, and I had a job and stuff myself).
Both kinds are OK. RPGs are a lot like any recreational sport. We don't think of it that way because we're geeks, but you might want to read my blog post RPGs as Sports: League vs Pick-Up Games for more. In recreational sports, everyone understands the difference between the various levels of commitment, ranging from "super serious these guys act like it's a pro team" to "pickup games on the local court." In RPG-land, due to Geek Social Fallacies we have issues setting these basic expectations, but we shouldn't.
I will note that I moved away from Memphis 13 years ago now, and that casual group is still meeting to this day! It's not "lesser," it's a different fit that works better for certain folks and their situations.
Now, you shouldn't mix your problem player in with the rest of this. That's a separate issue that has to be dealt with separately. There are other questions on this SE about how to handle that, and since I already mentioned my RPGs as Sports blog series I'll mention RPGs as Sports: Getting Cut which covers the topic.
For both situations - size and problem player - it is always possible to be polite. There's never any reason not to be polite. But politeness is about expressing yourself kindly, it doesn't mean you don't take action to make changes that need to be made. Making the hard decisions isn't "mean" and that's a bit of a psychological/social issue you will need to push through. Make changes to make things better while showing compassion to all involved.