Only a natural 20 (you roll the die and it comes up showing 20) is an automatic hit. From the SRD's rules for critical hits:
When you make an attack roll and get a natural 20 (the d20 shows 20), you hit regardless of your target’s Armor Class, and you have scored a threat.
Sometimes your threat range is greater than 20. That is, you can score a threat on a lower number. In such cases, a roll of lower than 20 is not an automatic hit. Any attack roll that doesn’t result in a hit is not a threat.
Your roll of an 18 was within the threat range of your weapon, but not an automatic hit. Once you rolled it, the next thing to do is take your roll of 18, add your modifiers, and compare the result to the target's AC. If your (modified) attack roll meets or exceeds the target's AC, you have hit, and with your rapier's expanded threat range, also scored a critical threat.
In that case, you would then roll again against the target's AC to confirm the critical hit:
To find out if it’s a critical hit, you immediately make a critical roll—another attack roll with all the same modifiers as the attack roll you just made. If the critical roll also results in a hit against the target’s AC, your original hit is a critical hit. (The critical roll just needs to hit to give you a crit. It doesn’t need to come up 20 again.) If the critical roll is a miss, then your hit is just a regular hit
However, if your (modified) roll of 18 wasn't high enough to hit the target's AC in the first place, you would miss.
Though it's not directly part of your question, it might also be useful to note (as KRyan does in the comments) that missing on a roll as high as 18 should set off some alarm bells. It might mean:
- Your target is way out of your league, and you should consider running away.
- Your character isn't very good at making melee attacks (you mention an attack penalty), and you should focus on improving your character or making attack types you're better at.
- Your group is making errors in attack roll calculations, and you should double-check to make sure you're not missing any modifiers.
If you keep running into enemies who are this difficult to hit, it might be worth looking into whether one or more of those apply.