What are the balance and game play implications of this house rule?
That this one, and only this one, source of disadvantage is special in that it cannot be countered by advantage.
So this is special.
It’s more special than, say, trying to hit someone while dangling by one hand from the end of a rope. Or while surfing. Or while prone. Or while launching ...