Only the first target would influence the attack roll
Against the first target, you either have advantage, disadvantage, or neither. You roll to hit them. The result of the roll is still one number, even if you applied advantage or disadvantage to determine that number.
From "Making an Attack" (cut down for brevity):
1. Choose a target. Pick a target within your attack's range: a creature, an object, or a location.
2. Determine modifiers. The DM determines whether [...] you have advantage or disadvantage against the target. In addition, [...] other effects can apply penalties or bonuses to your attack roll.
3. Resolve the attack. You make the attack roll. On a hit, you roll damage [...]
And from "Attack Rolls" (just to prove that an "attack roll" isn't the number on the d20, but the final outcome after applying modifiers):
When you make an attack, your attack roll determines whether the attack hits or misses. To make an attack roll, roll a d20 and add the appropriate modifiers.
Once that number has been determined, that number is your original attack roll. As per the text of the Sweeping Attack Maneuver:
If the original attack roll would hit the second creature, it takes damage...
Whether or not the second creature would have imposed advantage, disadvantage, or neither, is not taken into consideration. RAW, it simply says that if the original roll, meaning that one number that was the outcome of the first attack roll, regardless of how you arrived at that number, hits them, then they take damage.
Consider also things like Bardic Inspiration or other situational bonuses that your attack roll might have gained. If you had a Bardic Inspiration die and used it to hit the first enemy, clearly you wouldn't have been able to use it against your second enemy (if you were making two separate attack rolls, as per Extra Attack or something), and yet in this case, it would be considered part of the original attack roll, so it would still be applied to the second target.
This also fits the flavour of the Maneuver; it's called "Sweeping Attack", and you only make one attack roll. It implies that, narratively, you swing your weapon once and you go through one target into another, in one sweeping motion. Given that narrative (at least, that's how I imagine it would look), I wouldn't expect the second target to be that important with regards to the original attack roll.