No, it's not broken. The two points you highlight make sense and work like the following, but they're still only part of the story:
- Advantage for the attacker in the dark means “I, the target, have no way to notice when and where from I am attacked, which makes it harder to avoid.”
- Disadvantage for the attacker in the dark means “I, the attacker, cannot properly see my target in order to aim as accurately as I usually can.”
However, invisibility has another effect: not even knowing where the target is. So the target has the benefit of the attacker not being able to attack them at all unless they locate the target first:
An invisible creature is impossible to see without the aid of magic or a special sense. For the purpose of hiding, the creature is heavily obscured. The creature’s location can be detected by any noise it makes or any tracks it leaves.
If the target isn't revealing its location (i.e., by talking, or moving without Stealth checks), the attacker needs to do something specifically to locate the target or they can't shoot at them at all. This puts the target in the better position, because they can't be targetted normally. Only when the attacker knows where the target is does an attack roll become possible, and then yes, the advantage and disadvantage cancel out for the reasons above.