Unlikely, but may depend on your table
I've played a paladin whose focus was tattooed on his face for precisely this reason. At my table, component tracking isn't something we've really focused on, so we haven't seen any balance issues. However, if your table tracks components more closely it may be less balanced.
Sword and Board/ both hands occupied
There are also relatively few instances where the material component becomes a hindrance. If your character is wielding a sword/board combo, then this can be more of an issue. My paladin was a heavy weapon specialist and the rules do allow for a hand to come off the weapon which minimizes the mechanical requirements there.
But a character whose hands are each occupied with two different things may tip the balance of this. If not, then it's of minimal impact from my experience.
Disarming/taking from other characters
If your DM allows the Disarm optional rule or if they allow a mechanic for taking items from other characters there is more of a concern here - but that's easily circumvented by multiple pouches/foci.
Downsides
One downside to my decision to tattoo myself and mark my armor is that I'm quite recognizable. At times it's been a benefit, at others we've had to figure out a way to conceal me.