As far as I know, the multiverse of D&D 5E consists in different worlds all residing in the Material Plane, but the rest of the planes (Transitive Planes, Inner Planes, Outer Planes...) are shared between them.
Is there a lore explanation why each setting might have different deities? Since deities exist in planes other than the Material Plane, shouldn't those deities be the same in every setting?
D&D 5e relies a lot on older material from previous editions, particularly where cosmology is concerned. It makes explicit reference to the “multiverse,” and there are myriad suggestions that things are not supposed to be different (for most things) in 5e than they were before, but the details ultimately aren’t found in 5e and it will always be at least a little speculative how much still applies, since we know at least a few things are explicitly different (for example, 5e has the Weave be found throughout the multiverse, which it never was before and which conflicts fairly heavily with details of magic in other settings).
But still, the best source of details here are the Spelljammer and Planescape settings from AD&D 2e (and some 3e/3.5e for Planescape).
Spelljammer deals with the Prime Material Plane, which is subdivided into Crystal Spheres, which are effective solar systems encased in a giant, well, crystal sphere. Each crystal sphere has one or more stars, one or more planets, and on at least one of those planets, a campaign setting we’re familiar with. For example, the continent of Faerûn in which most of the Forgotten Realms takes place is on the planet Toril, which is found in a Crystal Sphere known as Realmspace. There is a separate Crystal Sphere called Greyspace, in which you’d find the planet Oerth, where the Greyhawk campaign setting is found. The Crystal Spheres float in a substance known as Phlogiston, and currents—known as Flows—sweep past them in predictable layouts that can be charted and so on. Spelljammers fly through the Flows to visit separate Crystal Spheres. This is the basic premise of the Spelljammer setting.
The important thing about Crystal Spheres is that they are governed by mysterious beings known as overdeities. For example, Ao of the Realmspace. For my money, it’s easiest to just say that overdeities aren’t deities at all, just something “over” them—because most of the rules of deities don’t apply to overdeities.1 Anyway, the point is, overdeities set the rules for their Crystal Sphere, including stuff like how magic and faith work, how gods are allowed to operate, and so on. Ao is known to have set quite a number of stringent rules in Realmspace, for example, and is also known to update those rules as he sees fit.
Separately from the Crystal Spheres—which are all found on one plane, the Prime Material Plane—we have other planes. Inner Planes of elements and energy, Outer Planes of alignment and belief, and Transitive Planes between them (Astral, Ethereal, Shadow, maybe Feywild). All Crystal Spheres are connected to the same set of other planes, and beyond the Crystal Spheres, overdeities have no say. So while Shar has to play by Ao’s rules in Realmspace, she’s under far fewer obligations in, say, the Outer Plane of Hades, which is where 2e put her divine realm, the Palace of Loss.2
And it’s the rules on the Crystal Spheres which explain the differing divine populations of each Sphere: not everyone wants to play by Ao’s rules, for example. Being worshiped on many Spheres is a good thing for a god, but there are a lot of Spheres out there and it’s not always worth it to try to expand to one particular Sphere, especially if the overdeity’s rules are against you. Moreover, expanding to a new Sphere is hard—even if one of your faithful gets there and manages to attract a flock, local deities can and will try to masquerade as you and receive that worship in your stead. (Shar is particularly infamous for this, which she has done successfully even against local deities.) Especially in Spheres with onerous rules—like Realmspace—local deities who are already established can have a nigh-insurmountable advantage. For instance, Shar is worshiped nowhere else, but in Realmspace she’s a huge deal—outside the Realms, plenty of other deities could tell her to sit down and shut up, but in the Realms she can tear nearly anyone else apart. Between Ao’s rules and the presence of powerfully-established local deities, Realmspace does not look like a good candidate for expansion.
The reverse is also true: Shar isn’t likely to be interested in starting from the bottom of the totem pole in some other Sphere. Dealing with Realmspace and her rivals there already occupies a lot of her attention, and expanding would be painful and possibly humiliating for her. So why should she? In theory, if she expanded a lot, it would substantially increase her power outside the Realms, but the Realms are what she cares about and while it would help there too, she’s already about as powerful as anyone can get there—there just isn’t a whole lot of growth potential for her that way. Better to try to tear down the few Realmspace rivals that are on her level (most notably Selûne). Expanding would be difficult and risky, for only marginal benefit.
The end result is that most deities avoid Realmspace (because the rules are difficult), and the deities who do interact with it mostly specialize heavily on it (because dealing with the rules often requires focus), so you get a mostly-unique population of deities in Realmspace. Other Spheres have easier rules, and so it’s easier for deities to be there as well as other Spheres, and you see more overlap. Greyspace is a lot like this, which is why you see gods there that are also found in many other worlds.
5e has—somewhat confusingly, both because the label seems a misnomer and because the label was used with a different meaning in previous editions—labeled these “greater deities.”
3e put the Palace of Loss in the Plane of Shadow, and 4e did something weird with the Towers of Shadow embedded in the Astral Plane—it’s usually best to ignore 4e for these discussions though, as its handling of canon was widely panned and largely reverted or ret-conned in 5e.
This is a decent question, but I believe your approach is actually upside down. Even on a single planet (like real Earth), there are multiple belief systems, multiple pantheons, a huge variety of religious practices and faiths. So the correct question to ask is not why all the different settings don't have the same deities, but how come you can place the realms of these varied deities into a single common cosmology.
The most likely answer is because players, as well as TSR/WotC wanted travel between campaign settings possible. So TSR put together a cosmology that places each realm into a common big multiverse. In particular, all the faith/belief matter was placed into the Outer Planes, organized by alignment in the Great Wheel Cosmology. Since the Outer Planes are infinitely large, realms/domains of all the deities of the D&D worlds could simply be fit. Details can be found here: Defining Realms, Worlds, Kingdoms, Planes, Dimensions and the Multiverse.
How do the in-game sages perceive this whole thing? There is speculation on this, with some sages proposing that the elements from the Inner Planes coming together to form the Prime Material Plane (with many crystal spheres) and the belief/faith of the beings in the Prime Material Plane forming the Outer Planes. You can read more in the following answer: Do gods need to be worshiped? (There is also speculation that like-minded deities (those of similar alignment) banding together against common enemies, hence placing their realms on the same part of the Outer Planes.)
While I'm not aware of any Fifth Edition product that contains an explanation, this topic was covered in some depth in earlier editions of the game. Specifically, the Planescape and Spelljammer campaign settings released for Advanced Dungeons & Dragons Second Edition both dealt extensively with multiple prime material plane worlds, and so needed to address this question.
The Spelljammer campaign setting stated that each prime world's solar system (or geosystem, in the case of Oerth) is inside a separate crystal sphere, and that a deity needs to have at least fifty followers in a crystal sphere to grant spells to priests in that sphere. Any priest travelling between crystal spheres, or in a crystal sphere containing few followers of that priest's deity, cannot be granted spells by that deity. (There was no confirmed explanation for why this should be so, but the books did speculate that it might be a natural property of the prime material plane that most or all deities have no ability to change, or that it might be the result of some ancient agreement between deities.)
The Planescape campaign setting went further, and explained that the culture of a deity has to match that of a Prime culture in order for that deity to be a good fit for that culture. As a consequence, deities naturally fall out of favour and are forgotten on prime worlds whose culture does not match their ethos, mythos, aesthetic, and technology level. According to the supplement On Hallowed Ground, this explains why the Babylonian pantheon is generally only worshipped by cultures that have only just invented agriculture, why the Greek pantheon is generally only worshipped by cultures resembling those of Hellenic Greece, and why members of the Greyhawk pantheon are only popular in cultures resembling those of the parts of Greyhawk in which they are worshipped.
The setting also mentioned the existence of "single-sphere powers," deities whose power is tied to just one specific crystal sphere, but who are significantly more powerful in that sphere than their number of worshippers would suggest. No explicit explanation was offered for this phenomenon, but such powers include the Forgotten Realms' Ao, Selûne, and Shar. (I suspect this exists purely so that the designers could get away with deities who were relatively powerful but also specific to a certain setting.)
Putting all of that together, we can surmise that single-sphere powers are unable or unwilling to spread their faiths beyond their native spheres; that other deities need to rely on missionary activity, worshipper emigration, or accidental cultural contamination to become known in new prime worlds; that these methods of spreading the faith are only likely to succeed if they're a good match for the new world's local culture; and that deities can eventually fall out of favour and be forgotten on a world if the culture shifts too much. Knowing that, it's not surprising that many deities are only worshipped in some crystal spheres.
There is no over-arching Dungeons and Dragons cosmology that can adequately answer your question. Each campaign setting is a distinct ontology with no relationship to any other. Any answer would necessarily be from the perspective of an individual setting.
Some settings - like Spelljammer and Planescape - are designed to encapsulate many possible worlds. However, they are also just individual settings and their lore does not necessarily apply to the lore of other settings.
For example, Al Qadim could be interpreted to take place on a material plane which is accessible to PCs in a Planescape campaign. However, nothing in the Planescape lore could be assumed to be true for the Al Qadim game - they are entirely distinct things.
