It's "ok" at any time.
Ideally however you would start with setting expectations - perhaps they will buckle down and come regularly if you say that's an expectation of your game (which is fair). Just booting them without setting that expectation is pretty much by definition unfair however, as there are different kinds of gamers who expect different things from a game. (See my blog post RPGs as Sports: League vs Pickup Games for more, and probably RPGs as Sports: Getting Cut for this topic in particular.)
In terms of other options, back when I was in Memphis I had a large group of gamers; some were casual players and others, including me, were looking for a more committed experience. And that's fine - like with sports, there are those who want a pick-up style game and others that want a league style game (see my blog post RPGs as Sports: League vs Pickup Games for more, and probably RPGs as Sports: Getting Cut for this topic in particular). Anyway, I ended up spawning off two groups - the "serious" group, which I would GM and for which I set clear expectations (weekly, only one miss a month tolerated, all in character talk no table talk, etc.), and the "casual" group, that was just whatever - and that group we rotated GMing in. So if there's a bunch of gamers and people have enough free time they maybe want multiple gaming opportunities, that's another way to approach it.