What distinguishes spells of level 1-5 from those of 6-9?

Question

The fifth edition of D&D is very particular about not spamming spells of level 6-9.

This is most obvious when it comes to warlocks, who have enormous freedom in what spells of level 1-5 they can cast and when, but who can only cast spells of levels 6-9 once a day. Their special features that give them more spells after a rest only work up to level 5.

Similarly, sorcerers can only use sorcery points for spells of level 1 through 5. Minor spell-casting classes only cast spells up to level 5. Monk's ki abilities only mimic spells of up to level 5. Druids' and clerics' circle/domain powers only grant additional spells up to level 5. A wizard's arcane recovery only works for spells of up to level 5.

Clearly the developers are going to extreme lengths to distinguish between spells of level 1-5 and those of level 6-9. My question is:

Why?

More precisely,

What shift in power occurs between levels 5 and 6 that makes it imperative to restrict access to the higher levels?

I've found some partial answers. For instance, no classes have spells that allow for transportation of a large group of people at once until level 6 (when players get Arcane Gate and Wind Walk). Also, level 6 seems to be when all classes get spells forming damaging barriers (Wall of Ice, Wall of Thorns, and Blade Barrier). But this only explains a small subset of spells.

A perfect answer would include why the jump from level 5 to 6 is more drastic than the jump from 3 to 4 or 4 to 5, for instance.

Answer 1

They "change the way adventurers interact with the world".

This is a direct quote from page 37 of the Dungeon Master's Guide where it describes the tier of play called "Masters of the Realm" (levels 11-16)

At this tier, adventurer's make their mark on the world...

The key spells that the Dungeon Master's Guide discusses are things like instant kills and other very swingy combat effects like massive healing and damaging barriers ("disintegrate, blade barrier, and heal") as well as spells that...

alter the way players approach their adventures

(spells like "word of recall, find the path, contingency, teleport, and true seeing")

These are spells that allow for rapid movement across large distances effectively, and the ability to deal with enemies with powerful resistances (like those with Limited Magic Immunity or ethereal creatures).

Answer 2

They are one of the main distinctions from Half-casters to Full Casters.

Paladin and Ranger stop at 5th level spells. Obviously, they have lots of other (martial) features, since they are not dedicated to spellcasting only.

Flavor-wise, this means that these spell levels are the results of someone dedicating his entire life to spellcasting, while Half-Casters are dedicating only part of it.

Tier-levels

As David Coffron mentions from DMG, we have tier levels. More than just flavor-wise for "how adventurers interact with the world", these tier leves are used in the Adventurers' League games.

From the 5-10 tier to the 11-16, the adventurers start to travel to other planes and are expected to get more powerful magical items. It is probably the greatest "gap" between tiers.

This gap means PCs are expected to get a decent power spike there. It applies to most classes, not only Spellcasters. Note that classes like Barbarian and Fighters get powerful features at 11th level. Fighter, for example, similar to the first tier break (5th level) where he gets his first Extra Attack, gets another Extra Attack now, which is a permanent \$50\%\$ increase in his DPR.

Note that I'm not trying to compare classes or features between classes, I'm saying "powerful feature" as "within the class features, this is a major one".