The DMG is the least useful of the core books
First, welcome to 5th edition D&D, the learning curve isn't too steep for this system and you should be able to pick it up fairly easily. Before I get to the main part of the answer there is something I would like to clear up.
Unlike other systems which publisher all the important rules in a single Core Rulebook or similarly named tome, D&D chooses to split its rules across the three main rulebooks. The Player's Handbook (PHB), Dungeon Master's Guide (DMG) and the Monster Manual (MM). In an ideal world a DM would have access to all three of these but some are more important than others.
First, you need the Player's Handbook. This book contains all of the core playable races, character classes, spells, feats and equipment. Without this book your players won't be able to make characters or look up their abilities. At my table it is by far the most used book of the three.
Second, I suggest getting the Monster's Manual. This may be contentious but I (and my friends that DM) find the MM more useful than the DMG. The MM contains the stats for all the creatures and enemies your party will encounter throughout the campaign. With just this book a good DM can run entire campaigns with very little preparation.
Finally, there is the Dungeon Master's Guide. The DMG is the book for all the other stuff. Rules, guidelines and DM specific advice that the players don't need to see. It has world and campaign building advice and bunch of extra rules and advice that you don't really need for running the basic game, but come in handy later. Personally I very rarely use the DMG at my table and only occasionally reference in between sessions. I firmly believe you can get by without owning it at all.
There is also the Basic Rules contain a reduced version of the three main books and can also be a good starting point. It contains just enough that you can decide if D&D 5e is the game for you, but not so much that you don't need to buy the books anyway.
What is the Starter Set?
The starter set contains:
- 64-page adventure book with everything the Dungeon Master needs to get started
- 32-page rulebook for playing characters level 1–5
- Pre-generated characters, with character sheet
- 6 dice
These are the absolute basics and everything you need to get started playing D&D 5th edition. See Difference between DnD 5e Starter Set and Basic Rules for more information.
So which should you get?
You seem to be deciding between the DMG and the Starter Set but didn't mention having access to the PHB. In my opinion it is more important to get the PHB and MM than the DMG. Since the Starter Set contains basic versions of all three, this is probably a good choice, particularly if you don't own the correct dice.
If you combine the Starter Set with the Basic Rules you should be able to play for a while without major issues. When you start wanting more options, first look into getting a copy of the PHB, then MM and finally the DMG and/or other expansion books.