This question has already been well answered, but I'm going to add a specific scenario. Honestly a one sized fits all starter set isn't a great idea because its trying fundamentally to fill a lot of different needs.
So here is a set I think is most important:
Beginner DM starter Set
Red Box
The Red Box was designed as a starter, and once person having it (usually the DM) is a good way to get started. It will be a bit simplistic however. But the immediate question as a DM will be where do I go from here?
Dungeon Masters Guide I and Monster Manual (or Monster Vault)
The first thing a newer DM needs is resources. You need ideas to draw upon, a foundation that you can build from, instead of working from scratch.
The Monster Manual is the go to place for Monsters, and therefore enemies.
The Dungeon Master Guide I has a lot of basic resources for things like magical items.
The Monster Vault is like a revision of the Monster Manual, but as a new DM don't sweat which one you get - whichever one you can get for cheapest will work fine.
Dungeon Masters Guide II
If you are a new DM, who has never DMed before, The 4e Dungeon Masters guide II remains one of the best guides for actually teaching you to be a better DM.
Its sections on:
- Group Storytelling
- Advanced Encounters (including encounters as story, pacing, player motivations, etc.)
- Skill Challenges
- Customizing Monsters
- Adventures
- And Paragon tier campaigns
Are among some of the best DMing advice ever printed. The mechanics in it, while not perfect, are good examples of what you can draw on to add extra levels of flavor to your campaign. Some ideas here, like Monster Themes - are only really now being picked up and explored almost a decade later in new games like ICON.
If you have been a GM / DM for other games, some of this advice has likely made its way to you, as other games have picked up tricks from it, but for a new DM, I highly recommend it.