The Superiority die is not affected by a critical hit because you only double the dice that are rolled against your first target
The section on "Critical Hits" states (emphasis mine):
When you score a critical hit, you get to roll extra dice for the attack's damage against the target. Roll all of the attack's damage dice twice and add them together.
The Sweeping Attack Maneuver states (emphasis mine):
When you hit a creature with a melee weapon attack, you can expend one superiority die to attempt to damage another creature with the same attack. Choose another creature within 5 feet of the original target and within your reach. If the original attack roll would hit the second creature, it takes damage equal to the number you roll on your superiority die. The damage is of the same type dealt by the original attack.
Note that with various other Battle Master Maneuvers the superiority die's damage is rolled twice because the damage from the die is part of the attack's damage roll.
With this Maneuver in particular the phrase "with the same attack" is used; you are not making a new attack, and so the damage from the superiority die is also part of the original attack's damage roll.
It certainly looks like you roll the die twice, at first glance, but on a closer reading, you'll note that the critical hit section says that you double the damage "against the target" (this is singular). The second creature you are hitting with this attack is a new target, different from your original target, and so the damage that comes from the superiority die is actually not affected by the critical hit.
This is slightly supported by Jeremy Crawford's now unofficial ruling on critically striking with the green-flame blade spell:
Q. If you critically hit with the attack part of Green Flame Blade, and are greater than level 5, do you roll extra dmg against the 2nd target?
A. The splash damage of green-flame blade isn't affected by the attack critting. Think of the attack as process X & the splash damage as Y.
Here he uses a different line of reasoning for why the damage is not doubled, but it is a similar situation.