Since rolling a number up to 20 with modifiers (an example 17 + 3) is not counted as a critical hit, what happens in the following case?
If I roll a natural 20 and because of penalties end up with a total of less than 20 (an example 20 - 3) does it still count as a critical hit? Or in this case would it resemble the natural 1 with positive modifiers taking you out of critical error?
Player's Handbook, page 194:
Rolling 1 or 20
[...] If the d20 roll for an attack is a 20, the attack hits regardless of any modifiers or the target's AC. In addition, the attack is a critical hit, as explained later in this chapter.
If the d20 roll for an attack is a 1, the attack misses regardless of any modifiers or the target's AC.
Pretty clear, right? If the die says 20, it's a hit and a crit, and I don't care if you have a -38 on the roll. And if you roll an 17 and add +7 to it, you didn't roll a crit even though your total was above 20. Only the number on the die matters.
As to your last line, I'm not sure what you mean about a "natural 1 with positive modifiers". As you can see by the above rule, a natural 1 is a miss no matter what bonuses you could apply to it (unless you have advantage, or some other way to roll again and use the number on the new die instead). D&D 5th Edition does not have any "critical miss/failure/error" rules, so those would be entirely homebrewed by your DM and I can't make any comment on how those would operate.
That said, there's a strong argument for not having critical miss rules. The higher number of attack rolls per turn from highly skilled characters means they'll roll critical misses more often than a low level character would, which seems rather silly.
If the d20 roll for an attack is a 20, the attack hits regardless of any modifiers or the target’s AC. This is called a critical hit, which is explained later in this chapter. (PHB 194)
All that matters is the number you roll on the d20. If that number is a 20, it is a critical hit, no matter what modifiers you would have normally had to the roll. If you roll a 20 and have a -50 modifier to the attack, it still counts as a critical hit.
To be clear, you only get a critical hit if you roll a 20 on a d20. If your roll is not a 20 then you don't have a critical hit, even if your bonuses would make the total attack 20 (or more).
The case of rolling a 1 is the same:
If the d20 roll for an attack is a 1, the attack misses regardless of any modifiers or the target's AC. (PHB 194)
So if you roll a 1, your attack misses, no matter what bonuses you had to the attack. Do note that D&D5e has no "critical error" rules by default beyond attacks missing when a 1 is rolled.
