We talked about random initiative in our group a few weeks ago. The DM mostly cited the "I do X because the boss can't act until you are done" kind of thing as a reason.
There are, however, a number of drawbacks. Here's what we could see:
It makes readied actions a big gamble
This is, in my opinion, the biggest sin of importing randomized initiative into 5e D&D.
Suppose I act after the monster in a round, and I want to ready an action to shoot an arrow at it as soon as it sticks its head out of full cover.
Unfortunately, the next turn I happen to end up higher on the initiative than the monster. My turn comes up again before my readied action has even had a chance to go off. I have, effectively, lost an entire turn.
This happens about 25% of the time with your proposed system. That may not sound like a lot, but given that the penalty is completely losing a turn, it's still kind of bad.
Why do I find readied actions so important? Because they are a huge enabler for teamwork and improvisation. If you can't ready actions effectively, you're a lot more likely to get locked into an "I attack it until it falls over" mindset.
It also makes ranged characters/monsters more powerful.
It messes with reactions
A superset of the above problem. Suppose I'm a counterspell-happy wizard fighting a spellcaster. Each turn, I use my counterspell to interrupt their spellcasting. This works fine with a static initiative.
But in a random initiative system, I would sometimes get back-to-back turns ("losing" a reaction), and sometimes have turns where the monster went twice in a row (causing me to be unable to counterspell).
This issue can be mitigated by having reactions (and some durations) reset at the beginning of a round, rather than the beginning of a player's turn. That's a little less intuitive, though.
It reduces, but does not eliminate initiative-order metagaming
Randomizing the initiative does eliminate a lot of player knowledge about the ordering of turns. But it doesn't eliminate it entirely.
You will still run into instances where two PCs have not yet gone, but the monsters have. In this case, the players know just as surely that they are clear to proceed as they would with a pre-set initiative.
It discourages pre-planning
It's one thing to say "okay, I'm up in two turns. I doubt much will change, so I'll start figuring out what I want to do."
It's another thing to say "okay, some unknown number of people are going before me so... I'll start planning ahead?"
It's also a lot easier to go "Bob just went, time to start thinking about my turn."
It adds per-round overhead
This one doesn't apply much to you, since it will generally not take long to gather up the tokens, put them in the bag, and give it a good shake.
Still, randomized initiative does usually take a similar amount of time to establishing a pre-combat initiative order. Except it takes that amount of time every round.
In your system, you could probably save even more time by just doing the draws at the start of combat, setting them in order on the table, and looping through that.
It makes hard combats "swingy"
A challenging monster (like a dragon) getting two consecutive turns can be pretty back-breaking (the worst case scenario being a player taking two breath weapons and dying before they have a chance to react). In the other direction, multiple PCs getting double turns can make a fight dramatically easier, potentially spoiling a boss fight.
Boosting initiative is a relevant "thing" in this edition
There aren't a ton of classes who are destroyed by eliminating initiative modifiers, but there are a few. Assassination rogues are the ones most hurt by this change. Also, as noted elsewhere, barbarians, the alert feat, and any heavy or medium armor wearers who put points into dexterity for the initiative bonus.
It's also worth noting that classes that rely on buffs (such as Druids) will tend to want to be able to invest in a higher initiative, although they do not always do so in practice.
Summary
All in all, none of this is game breaking. I don't think that I would use this system, as generating a turn order doesn't take our group very long (just a quick pass around the table, asking what each person's numbers are).
Still, if your table is running into issues, go for it. Just allow people to rebuild characters after implementing it.