Are there any rules or penalties for trying to roll stealth while for example carrying a torch through a dark cave? It seems like it would be basically impossible for anything not to see you, but I couldn't find any rules for this.
Yes, there's rules.
If you're carrying a torch, you can't Stealth, and it's really easy to see you. Why? Per the Vision and Light rules, a torch casts "normal" light 20 feet out. You can't use Stealth in normal light. Relevant rules, emphasis mine:
It casts less light farther out than that, and anybody can of course see that pool of light moving around so even if you could Stealth you're not really very well hidden.
Now if a party member is holding the torch and you're 30 feet behind the group, then you could attempt it.
The rules for Stealth are not written in a legal-contract form, but instead as a comprehensible description of what you need to do to be able to use the skill.
Effectively, if someone can see you and you don't have a hiding spot, you can't use Stealth against them.
If you're standing in the middle of a dark cave holding a burning torch, it is not possible to avoid being observed visually, and therefore you cannot use Stealth to evade detection by a vision-using creature. You could use Stealth if you had some cover to hide in or behind, but that only prevents you from being observed directly – if your torchlight is still visible, that's ample reason for someone or -thing to go investigate your location, just like making a bunch of noise would also reveal that something of interest is hiding behind that boulder.
In conclusion, don't try to hide while carrying undoused light sources in dark places. It doesn't work unless what you're hiding from is blind.
While I appreciate that from a visual point of view, hiding while holding a torch would be silly, I think it important to point out that there is nothing stopping someone from trying to be stealthy while holding onto a torch -indeed, they are two distinct yet related things.
Sound plays just as an important factor into your Stealth check: sure if creatures are alert and in the path of your torch, then forget about it, but it will be your loud footsteps and unconcealed clanging armor that draws creatures from connecting areas.
As per in real life you should still be able to do a Stealth check without penalties (apart from armour and terrains ones) and indeed, attempt to be stealthy, but if you are within visual range of something, the check automatically fails with that creature. Hence as already suggested above, best option would be to have say a forward scout with darkvision or similar -and if that isn't possible, then all should be stealthy and pray that you detect enemies before they detect you. Hooded lanterns -if money can allow- are always a better choice specifically for these situations.