The answer to this is the combination of a few different rules, most which are found in the Player's Handbook ("PHB"), with a few extra notes found on page 77 of Xanathar's Guide to Everything ("XGE"), because these sort of questions did come up after the initial rules release.
First off, XGE makes its clear that when you're asleep, you have the unconscious condition (PHB p.292). So among other effects,
- An unconscious creature is incapacitated, can't move or speak, and is unaware of its surroundings.
- Attack rolls against the creature have advantage.
- Any attack that hits the creature is a critical hit if the attacker is within 5 feet of the creature.
(The incapacitated condition merely says you can't take actions or reactions.)
Furthermore, there's surprise involved here (PHB p.189):
If you're surprised, you can't move or take an action on your first turn of the combat, and you can't take a reaction until that turn ends.
The rules given in XGE expand on this; in brief:
- A sleeping creature wakes up if they take damage, if somebody uses an action to wake them by physical means, or if there is a sudden loud noise.
- Your passive perception is not reduced for being asleep, and your passive perception determines how sleep-disruptive nearby sounds are (such as people talking or whispering).
So to pull it all together...
- The Rogue's stealth roll is opposed by the goblin's passive perception.
- Everyone in the fight rolls initiative, as usual. The unconscious condition doesn't say you can't roll initiative.
- The goblin is unconscious until something happens to change that state, such as taking damage, and is surprised during their first turn in combat, whether they are awake or not at that point.
If the rogue goes first in initiative order and attacks the goblin in melee, they get advantage, sneak attack damage, and an automatic critical hit (if the attack hits). The goblin wakes up (assuming they aren't just dead at that point), but still can't take an action on their turn because they are surprised; but after that do-nothing turn (essentially spending their whole turn becoming un-surprised), they can take reactions, as they are no longer unconscious or surprised.
If the sleeping goblin won initiative, then it would be unable to act on the first turn (due to the unconscious condition); but after taking damage they would be awake and not surprised, thus able to take reactions immediately and able to act normally on their next turn.
If the rogue's stealth check had failed, then the goblin would wake up due to some noise, it would be up to the DM to rule whether the goblin is still surprised (most likely yes), and it would proceed much the same as if the rogue had failed stealth against any other target.
The DM might modify this.
It might seem odd that a sleeping creature is no easier to sneak up on than the same creature awake but inattentive (reading, talking, eating, or just standing around, for example). The DM might decide to give you advantage on your stealth roll against a sleeping creature to take that fact into account, but that isn't actually in the rules (which is fine, that's why we have a DM and not a Game-O-Tron 2000).
The DM might decide that the surprised effect doesn't start until the goblin wakes up, so even if it won the initiative roll, it would still have to go through one round of being surprised after being awakened by damage. That's not technically how surprise works, but it certainly makes sense in the context of this situation.