I don't believe that a Rogue can normally sneak attack when it grapples. This is why (let me know if I'm wrong):
Under Sneak Attack in the Rogue section it states
Once per turn you can deal an extra xd6 damage to one creature you hit with an attack if you have advantage on the attack roll. The attack must use a finesse or a ranged weapon.
In Chapter 9 under Grappling it states
can use the Attack action to make a special melee attack, a grapple. If you’re able to make multiple attacks with the Attack action, this attack replaces one of them.
So this to me is a special attack not a finesse or ranged weapon attack since it requires Athletics(Strength) to engage, and one free hand not holding a weapon, therefore Sneak Attack is NOT allowed imo)
You would also need the feat Grappler from Chapter 6 which states
You have advantage on attack rolls against a creature you are grappling […]
So you apparently don't have advantage on attacks without this feat, so you can't use Sneak Attack.
So my conclusion is that a Rogue can only use sneak attack in a grapple if
- it has the Grappler feat, as well as
- is multi-classed with something like fighter/ranger/monk/etc. where at level 5 or so it gets a second attack, since grapple is a special (strength based) attack.
I am curious whether this works. I would love to make a strength-based monk 6 / rogue 1 to focus on a one-two grappling-then-assassinating tactic: one attack is to grapple and the second is to sneak attack. This would be a great way to get Sneak attack every round. Please correct me if I'm wrong in my understanding.
One last thing. If two creatures are in a grapple you just make the check at the beginning of the next turn so the lvl6 Monk/lvl1 Rogue could have two attack (one of course being sneak attack)? Or would you have to grapple attack again, and get one attack (finesse, w/sneak attack)?