The premise of your question is somewhat incorrect. Level 20 is a standard progression limit only in 3e and 5e.
In First Edition AD&D, there is no level limit. Specific class advancement tables describe advancement from anywhere from 29 levels (cleric) to 9 levels (fighter) but only for purposes of showing how high certain abilities can go, they all note you can go on up infinitely from there. The big differentiation is "name level," which is usually in the 9-10 range, where the character stops getting as large level advances in hp and starts focusing on building kingdoms and whatnot.
In BECMI D&D (Red Box Basic), you can go up to level 36, and there are level breaks from Basic (up to level 3), Expert (up to level 14), Companion (up to level 25), Masters (up to level 36), and Immortal (past that, cashing in XP for power) with differences in those levels of play.
In Second Edition AD&D, advancement is described on convenient charts up to level 20 but there is no limit, with a breakpoint at level 9-10 where you stop getting full hit points with each level. It has a section in the DMG about how play gets harder to be satisfying at higher levels and that you probably need to shift campaign styles. In terms of playstyle recommendation past 20 it says
Consummate skill and creativity are required to construct adventures for extremely powerful characters (at least adventures that consist of more than just throwing bigger and bigger monsters at the nearly unbeatable party). Very high level player characters have so few limitations that every threat must be directed against the same weaknesses. And there are only so many times a DM can kidnap friends and family, steal spell books, or exile powerful lords before it becomes old hat.
It then recommends retirement as an endgame.
In Third Edition D&D, advancement is described up through level 20, with levels past that described in "an upcoming rulebook." It was lightly treated in the DMG but then more fully in the Epic Level Handbook in the 3.5e days. 3.5e and Pathfinder are lightly changed derivatives of 3e by design and so aren't really different editions with different ideas driving them as far as this goes. The Epic Level Handbook describes its intention, which is to change playstyle from the level 1-20 model to being legendary, allowing PCs to "wield powers that other characters (even 20th-level ones) can only dream about." It notes that PCs may have had the thrills of running nations and political machinations come and go and this is their gateway to discovering the secrets of the universe, plugging into the primal cosmic battles, etc.
You may also want to review the 3.5e DMG's discussion on epic characters and why attack and save bonuses cap out at 20 on p.207. (Summary: too many attacks causes slog and too much disparity between faster and slower save and BAB progressions causes balance issues). Also on p.210 they explain that many classes have been balanced assuming that 1-20 progression and that balancing classes for infinite progression is way harder.
In Fourth Edition D&D, the level limit is 30. There is no implied "soft cap" at 20. Play is grouped into rough "tiers" from the heroic (1-10) to paragon (11-20) to epic (21-30), but it is a continuum.
In Fifth Edition D&D, the game describes four tiers of play (1-4, 5-10, 11-16, 17-20) with "epic boons" available after level 20.
From this, you can take away several things.
The nature of play changes with level. Kicking down the door and killing something works well as a low level adventure and less well as a high level adventure, due to both repetition and the powers and abilities available to higher level PCs and foes, so shifting to more political or grander-scale adventures becomes desirable. Some editions formalize this with tiers, others say "low level, high level, and very high level", etc. What level that is varies by the specific D&D edition and its core rules.
There are a variety of "soft caps" and "hard caps" across the editions - tier boundaries, name level, etc. Only in 3e and 5e is 20 specifically a meaningful number that one might describe as a levelling cap (with later progression options). These numbers are not based on some arcane math but on when the designers feel like gameplay breaks down under its prior level paradigm. 4e is reusing the "epic" term but there is a continuum from 1-30 where epic can't be considered a meaningful cap, even a soft one, it's a breakpoint like the one at level 10. It's basically just using previous edition words for that level band to comfort people.
"Epic Level" play is a 3e thing based on a very specific product and terminology coined for 3e. Most references you've seen to "post-20 play" and "epic" are just an outgrowth of 3e play specifically. You are seeing something "across all of D&D" which isn't really across all of D&D.
Since versions of D&D mostly share certain rule similarities, the breakpoints of power - mostly cemented by what spells become available (fly, teleport, wish, etc. change the dynamics of the game by their availability) tend to be in around the same spots. So short of devising new things (tenth level spells, epic powers, etc.) versions of D&D that use the traditional spell advancement of "a new spell level every couple character levels" cap out spell power right before 20, where then it becomes a game of "more, but not really different" without additional rules that are pointless to include in core books where 99% of people don't ever get up to level 20 anyway. But this means that the around-level-20 breakpoint isn't really deliberately designed, it's more of an inevitable endgame of the spell system, unless you deviate from it (as 4e did). Even in BECMI, the Master rules (level 26-36) are only 32 pages long and are basically some new spells and then siege engine rules. When I played Basic no one ever went past Companion because the game got pretty weird and uninspired there.
Given a class-and-level system of D&D's kind, and the kind of Vancian casting powers traditionally available at levels around 5 (fly, fireball), 10 (teleport, raise dead), 18 (wish, miracle) then you get a similar need to change playstyles at those milestones, with 1-5 being your gritty stuff, 6-10 being (super)heroic, above that needing to change more to political and larger scale concerns to keep challenge and interest, and around 20 becoming a point of diminishing returns where you need to do something different to maintain challenge and interest given how spells etc. cap out there. BECMI Master pushed this past to 36 and got boring for that whole range; 4e went to 30 by discarding the Vancian tradition other editions share.
So it is incorrect to say that 20 is a soft cap across most editions, but this is the reason behind it in 3e/5e and the other "caps" and "breakpoints" and "tiers" in other editions in general. It's an emergent condition of the kind of ruleset D&D is and its historical trappings (Vancian magic being the most important) driving a change in playstyles at certain power inflection points. The designers explicitly talk about this in each edition's books regarding high/various level play.
If it needs to be stated more simply, 20 is not a magic number, it's just when having 9th level spells gets old.