It depends on how often you get to surprise
If you play up to level 10, the features that dominate how useful a subclass is are those you gain on level 3. They will be with you for most of your campaign, while the features you gain on level 9 will impact play only for a short time, at the very end. So let's focus on the level 3 features1.
I'll assume you have maximized Dex with a +3 bonus from your ability score. I'll also assume that you hit your opponents on average with a 65% chance, so that advantage will add 22.75% to hit chance. In adventurers league, characters have access to a magic +1 item starting with tier 2 (thanks to @stevenjackson, who also provided valuable input on how to best calculate Sneak Attack), and the tables in the DMG and Xanathar's also suggest you might have one by then, which increases to hit to 70% and dampens advantage to 21%. Since this covers the bulk of the time spent with the subclass (5 out of 8 levels), I will assume you'll have that. Your sneak attack bonus over these levels ranges from 2d6 to 5d6, I'll use 3d6.
Overall if you have a group that often is able to surprise the opponents, Assassin with two-weapon fighting is the best option, and maximizes the effect of your Surprise Attack and Sneak Attack. Soulknife is a bit better for reliable damage output, if you have a hard time surprising opponents and have no access to magical weapons.
You can expect to deal 23 more damage in a typical fight with Assassin than with Soulknife if you get surprise and have a magic weapon, and only 6 less, if you don't have either. That is, if you expect to surprise your opponents in 1 of 5 fights or more often, then Assassin is your best choice.
(Details below)
Bugbear's Surprise Attack
You are taking bugbear for the flavor, but it also happens to be one of the races best suited for maximizing melee damage in a round, due to Surprise Attack, which adds +2d6 to each attack it the opponent has not take a turn yet, and even works on multiple attacks.
However, be careful, because what counts is that you win initiative, not if creatures are surprised, no matter the feature's name. The effect of surprise is:
If you're surprised, you can't move or take an action on your first turn of the combat, and you can't take a reaction until that turn ends.
You still get to roll initative and take your turn, you just cannot do anything with it. So if the creature wins initative against the bugbear rogue, and than suffers through its inactive surprise round, it has taken a turn before the rogue, and the rogue will get no extra damage.
The average Dex of creatures up to CR10 is 13, so they will have +1 to initative on average. You have between +3 and +5 (if you maximize Dex), so lets assume +4 on average. That means you'll win about 66% of your initatives and be able to use the feature. The Swashbuckler Rogue has Rakish Audacity, which allows you to also add Charisma. Assuming you also have at least a +2 in Charisma, you could win 73.75% of those, for an extra roughly 8% of cases.
On average fights take 3 rounds (see also here), so if you can spring Surprise Attack that will affect 1/3 of your attacks. I think 8% of 1/3 of your attacks on just the Surprise Attack damage is not worth picking the subclass when you want to max damage.
If you manage to surprise a lot: Assassin
How often this works will depend on how your group plays -- do you have a wizard or druid that uses magical means to scout ahead, to help you maximize the chances to spy upon opponents and ambush them?
If you are ambushing a lot, then Assassin is the best choice. Its Assassinate feature gives you Advantage on attacks against any creature that has not taken a turn yet, matching perfectly to your Surprise Attack, and you automatically get criticals on creatures that are suprised, doubling your damage from the weapon, Surprise Attack and Sneak Attack dice.
First round. Raw Damage: 1d8 (Rapier) + 3 (Ability bonus) + 1 (weapon) + 1d8 (Assassinate) = 4.5 + 3 + 1 + 4.5 = 13 damage, to this you add (2d6 Surprise Attack + 2d6 Assassinate) x 66% = 9.24 from Surprise Attack, total 22.24 damage.
To this you add a two-weapon fighting attack with a dagger using your bonus action. You don't get your ability bonus on this so it is 1d4 (Dagger) + 1d4 (Assassinate) + 66% * 4d6 (Surprise Attack with Assassinate) = 2.5 + 2.5 + 9.24 = 14.24.
Your to hit chance is 70% + 66% * 21% (Advantage, wich you get when you won initiative) = 84% with your main weapon, 65% + 66% * 22.75% = 80% with the dagger.
Sneak Attack adds 6d6 with Assassinate, or 23 damage. Since you can hit on either attack, the chance to connect with it is 97%.
Adjusted expected damage: 84% * 22.23 (main) + 80% * 14.24 (dagger) + 97% * 23 (sneak) = 52 damage.
For subsequent rounds, you just get your normal attack of 1d8 + 3 + 1 = 8.5 on your main weapon, with 4.5 of it for criticals. You get 2.5 on your dagger, all good for crits. Your Sneak attack chance is unchanged but damage is half as you don't have Assassinate.
Adjusted expected damage: 70% * 8.5 + 5% * 4.5 (main) + 65% * 2.5 + 5% * 2.5 (dagger) + 97% * 11.5 + 5% * 11.5 (sneak)= 20 damage (19.6 to be exact).
Without magic weapon, this would reduce to 65% * 7.5 + 5% * 4.5 + 65% *2.5 + 5% * 2.5 + 96% * 11.5 + 5% * 11.5, or 18 damage.
Over three rounds you deal 52 + 20 + 20 = 92 damage. If you don't get to surprise, you only would deal 60 damage, and without a magic weapon, only 54.
Because winning initiatve is strong here, taking a feat that boosts your intiative roll like Alert might be useful to max damage, but it is likely better for overall playablitiy if you just maximize Dex to 20 with your ASIs and increase your to hit, damage, Dex skill checks, Dex saves and Initiative rolls .
For reliable damage: Soulknife
Soulknife has the Psychic Blades feature, which if you are empty handed allows you to make an attack with a base d6 damage, and an additional one with a base d4 damage as a bonus action, when you take the attack action. This effectively doubles the number of attacks you can make as a rogue, and unlike two-weapon fighting, it allows you to add your ability modifier to the damage both times. You cannot benefit from a magic weapon however. Your raw damage output per turn would be:
First Round Attacks: 1d6 (Psychic Blade) + 3 (Dex) = 3.5 + 3 = 6.5. Raw Bonus Action Attack: 1d4 (Psychic Blade) + 3 (Dex) = 2.5 + 3 = 5.5. For Sneak Attack you don't have a magical +1 to hit on the first attack and no Assassinate, for 96% * 11.5. Surprise Attack adds 2 x 2d6 or 14 damage in 66% of cases, or 9.24 damage.
Total Expected Damage: 65% * 6.5 + 5% * 3.5 (first) + 65% * 5.5 + 5% * 2.5 (second) + 96% * 11.5 + 5% * 11.5 (sneak attack) + 9.24 (surprise attack) = 29 total
The followup rounds 65% * 6.5 + 5% * 3.5 (first) + 65% * 5.5 + 5% * 2.5 (second) + 96% * 11.5 + 5% * 11.5 (sneak attack) = 20 total (19.6 to be exact).
You would deal in total for a 3-round fight 29 + 20 + 20 = 69 damage. If you don't get to surprise, you will deal about 60 damage.
Addendum: Maximizing Sneak Attack
One prominent feature for the rogue to add damage is Sneak Attack. There are many ways to get Sneak Attack, the simplest is when someone else threatens the creature by being within 5 feet of it while you have no disadvantage. For the calculations, I assumed you'll have such an ally, and will be able to get Sneak Attack, although that of course is not always the case. Independent of subclass, picking features that allow you to attack as a Reaction out of your turn allows you to get a second attack with Sneak Attack, and may help improve your damage output.2
1
Your damage output evolves over time, and Sneak Attack damage will make an increasing share compared to other sources. As baseline, I'll compare the result on level 3, when you gain the features.
2 One feat that allows you to do so is Sentinel, which you can pick up at level 4. It has other powerful benefits, such as stopping the creature if you hit it.
When a creature within 5 feet of you makes an attack against a target other than you (and that target doesn't have this feat), you can use your reaction to make a melee weapon attack against the attacking creature.
