Going by RAW, there are no drawbacks
The cloaks does what it says, and give the wearer advantage when trying to hide and disadvantage to those trying to perceive them.
As a DM, you can create drawbacks
Mostly centered around the phrase, "the cloak's color shifts to camouflage you."
So all day long, the wearer is fading in and out of perception via shifting colors to match the surroundings. This is fine if you're hiding in shadows (lots of dark greys) or spelunking in caves (lots of dull greys with patches of green here and there).
But try walking through town, or the corner store, and suddenly the wearer is surrounded by different colors. The cloak, in an attempt to keep the wearer hidden, becomes a calliope of swirling colors that shift as they round corners, talk to a vendor, or pass random NPCs on the street. I imagine the effect to be quite nauseating.
It's also a sure fire way to get the guards, shop keeps, and other people to pay close attention to the party. If they feel they need to stay hidden all day long, they are likely up to no good.
I've never run a game where the owner of the Cloak kept it up all day long. Generally, it was, "I'm about to go scouting, I'm putting up the hood." And, "I'm reporting back to the party so I take down the hood."
However, should someone try, I would let them know that the constant shifting of colors becomes a distraction to other party members. If they are standing still, the colors will quickly settle. But if they are walking, after some distance (measured either in time walking or distance walked) the rest of the party would suffer disadvantage on Perception checks (including -5 on Passive Perception). Once they stopped walking, the disadvantage would go away.
This still meshes with the cloaks natural function:
- If hiding, they are standing still or at least generally moving slowly. So they would still have advantage while trying to Hide.
- If you're standing still, the Cloak gives disadvantage to seekers as the wearer will blend in. If moving around, they are still hard to see and would still have disadvantage at seeing the blurry wearer. The observer disadvantage doesn't kick in immediately so that would not be a concern during combat for instance.
So in practice
- Walking through a dungeon, while the wearer is harder to see, the party is less likely to notice an ambush
- Strolling through town, the wearer would be hard to track, and because the tracker is suffering from disadvantage, they couldn't even focus on a different party member. But by the same token, the party would have a hard time telling if they were being tracked or not.
- In a shop, if the wearer is walking around to be a distraction, the shop keeper will have a hard time seeing if someone tries to steal something. On the other hand, they will also be on guard so would likely cancel out the disadvantage. It would have been better to just hide.