Losing attunement will prevent having disadvantage.
The following passage is from the attunement rules in the Dungeon Master's Guide (p. 138):
Without becoming attuned to an item that requires attunement, a creature gains only its nonmagical benefits, unless its description states otherwise. For example, a magic shield that requires attunement provides the benefits of a normal shield to a creature not attuned to it, but none of its magical properties.
The tense of the first sentence could be ambiguous ("becoming attuned" sounds like a one-time requirement), but the example clarifies the meaning: while you are not attuned to a magic item that requires attunement, you can't benefit from its magical properties, even if you were attuned to it in the past, since it's the current attunement state that matters.
So even if you were attuned to the oathbow once before, you cease to benefit from any of its magical properties when you are no longer attuned to it, treating it as a mundane bow from then on. Since you no longer benefit from having a sworn enemy and the bow is treated as mundane, there's no reason you would have disadvantage on attack rolls with different weapons.
In addition, once you lose attunement, you lose (not suspend or defer) your sworn enemy, so regaining attunement later does not reinstate that sworn enemy. Rather, when you regain attunement, you start with a clean slate without any lingering benefits from a prior attunement.