I'm very new to this game and am wondering, If the enemy has say 30 AC with all his equipment and buffs, what would I have to roll to hit him since I have to roll a 20 sided die to determine if I hit or not? Since 20 sided dices obviously don't go to 30 I'm curious as to how this work.
Attack bonuses are your friend.
Based on how you're attacking your target, various bonuses will be added to the roll.
If a spell, your spellcasting modifier and your proficiency bonus (PHB p.205). If a weapon, either your strength or dexterity modifier (based on the weapon and, possibly, your choice) and your proficiency bonus if you're proficient with that weapon (PHB p.194).
Then there are various circumstantial bonuses that might apply: bardic inspiration from your friend, the effects of a bless spell, the enchantment-based +2 of a magic weapon, a fighting style, &c.
30 is hard to hit, no bones about it. But a bless (2.5 avg bonus) and a 5th-level bardic inspiration (4.5 avg bonus) and a +1 weapon and your level 5 proficiency bonus of +3 and an ability modifier of +4 gives you a +15 attack bonus; you'll hit 30% of the time. Not great, and it takes some resources to get there, but it took the target some resources to get to 30, too.
So are critical hits.
Remember that a 20 on an attack roll always hits, regardless of AC. That may seem a stretch, but advantage and luck dice can significantly improve those chances. From 5% to ~10%.)
2\$\begingroup\$ A bit of cheese with Luck Dice can be had as well if you're going by RAW. If you give yourself disadvantage (by say, going prone) you can use a Luck point to roll an additional die and then choose among all three, giving yourself a 15% chance to crit. This is broken and any DM in his or her right mind will have nerfed this part of Luck. \$\endgroup\$– Rob RoseOct 11, 2017 at 15:41
\$\begingroup\$ @RobRose RAW is RAW - the Luck + prone strategy ought to be called the "Hey Baby Crit me one more time!" And this is how a legion of Flumphs killed the Tarrasque in 5e! \$\endgroup\$– SenmurvJun 11, 2021 at 22:39
Another option is do not hit them
There are spells within DnD 5E that do not require you to make a hit roll. Any examples I can think of instead require the target to make a saving throw.
The DC to resist one of your spells equals 8 + your spellcasting ability modifier + your proficiency bonus + any special modifiers.
A hail of rock-hard ice pounds to the ground in a 20-foot-radius, 40-foot-high cylinder centered on a point within range. Each creature in the cylinder must make a Dexterity saving throw. A creature takes 2d8 bludgeoning damage and 4d6 cold damage on a failed save, or half as much damage on a successful one. Hailstones turn the storm’s area of effect into difficult terrain until the end of your next turn.
At Higher Levels. When you cast this spell using a spell slot of 5th level or higher, the bludgeoning damage increases by 1d8 for each slot level above 4th.
Thus, you no longer need to overcome a high AC.
Note that it's quite likely that any creature that has 30AC is likely to also have decent bonuses to saving throws. Sometimes they will have a particular type to which they are susceptible.
For example, an Ancient Red Dragon has a natural 22AC, and a +7 to dexterity saving throws. So you overcoming it's natural AC might be difficult, but it failing a save (or in the example above, taking half damage even if it makes its save) is quite plausible.
4\$\begingroup\$ There's also the option to use attacks that don't need to hit AC, nor require a save. For example, drowning the Tarrasque in a pit it can't climb out of, or removing all the breathable air from an area. \$\endgroup\$ Oct 10, 2017 at 6:51
\$\begingroup\$ There's always Magic Missile, though it might take quite a few castings... \$\endgroup\$ Oct 10, 2017 at 21:33
\$\begingroup\$ @thedarkwanderer Obligatory link to Seven Ways to Kill the Tarrasque at Level 13 or Less. It's for 3.5e I think though. \$\endgroup\$– Rob RoseOct 11, 2017 at 15:34
\$\begingroup\$ @RobRose Actually it's 3e, except most of the information isn't quite right. It's the right idea, though. \$\endgroup\$ Oct 11, 2017 at 20:07
ACs don't get that high.
In the DMG (pg. 274), there's a big table that lists offensive and defensive stats by CR. In that table, the listed AC for a CR 30 monster is only 19.
The Tarrasque, the highest CR monster in the game, has an AC of only 25. The runners up, the Ancient Gold and Red Dragons, have ACs of 22. Hence, it's very unlikely that you'll run up against an AC of 30, even at high levels.
Bonuses get you there
As you level up, you'll accumulate abilities that will give you a bonus to your attack rolls, such as higher ability scores and proficiency, as well as things like Bardic Inspiration or magic items. All of these things add flat values to your dice roll, and thus make hitting ACs over 20 feasible.
For example, 1st level fighter, with a strength of 16 (+3) and a proficiency bonus of +2, has a +5 to hit. If she rolls a 17 on the die, she adds 5 to it, making a total attack roll value of 22. That means that she only needs to roll a 17 or higher to hit an Ancient Red Dragon, so she has a 20% chance to hit without any other bonuses. Of course, dealing enough damage to be more than an annoyance is another story.
11\$\begingroup\$ Being an annoyance to an Ancient Dragon tends to be a very brief experience. Potentially exciting, but brief... \$\endgroup\$– ShadurOct 10, 2017 at 9:57
4\$\begingroup\$ @Shadur: "Do not meddle in the affairs of dragons, for you are crunchy and taste good with ketchup." \$\endgroup\$ Oct 10, 2017 at 20:21