The only published rules were on the second edition of the game
Neither 5th edition nor 4th actually published any rules for the encounters with the Dweller at the Threshold. The only rules iv ever read about it were in the 2th edition sourcebook, Magic in the Shadows (p. 92), which say the following:
To begin an astral quest, a character must astrally project to a metaplane. Initiates able to use astral projection can do so on their own. Others must gain access to the metaplanes through the astral gateway power of a free spirit (see Powers of Free Spirits, p. 116). Either way, the character must project directly to the metaplane.
If several people want to go on a quest together, they can start from different places as long as they begin at the same time with the intention of traveling together. A magician in Manhattan can accompany a colleague from Seattle and another in Tokyo on an astral quest as if they were in the same room
Every astral quest begins the same: with the questor floating in the dark void where the Dweller on the Threshold lives.
The Dweller guards the metaplanes. Some traditions say the Dweller represents the traveler’s shadow, or dark side, while others believe it is a powerful spirit guardian of the gate between worlds since time immemorial. Whatever the case, the Dweller always requires a test before a traveler can enter the metaplanes.
The Dweller has no set form, varying in appearance and attitude from meeting to meeting. Generally, the Dweller’s guise is somehow relevant to the quest or the questor. An enemy, a loved one, or a deceased comrade are all possible forms, as is an appearance and manner designed to foreshadow things to come.
The Dweller also knows everything about an astral traveler: every crime, every secret, real name, numbered bank account, everything. If you go on an astral quest with companions, be prepared to hear the Dweller announce some choice tidbit of best-left-unspoken information. The Dweller plays fair, of course, revealing something about every member of the group. It is impossible to get around this by projecting into the metaplanes separately and joining up after passing the Dweller. Everyone must begin the journey together, and if a character is hurled back to his body, he cannot rejoin the quest.
After airing the travelers’ assorted dirty laundry, the Dweller requires each to pass a test of some kind to prove their worthiness. While this test should be roleplayed for maximum impact in the adventure or campaign, in game terms, each character should make a test against a target number equal to the Quest Rating using a skill or Attribute chosen by the gamemaster based on the purpose of the quest.
And finally, what happens if the character fails the test:
If the character fails the test, the Dweller sneers and the
quest fails. If the character succeeds, for every 2 successes add
an extra die to the character’s Karma Pool for use during the
astral quest; when the quest is over, those dice are gone.
Yeah, the quest fails, no details were ever given. So we can only assume you float back home and try again another day. Since this is mostly a roleplaying part of the game, the gamemaster is encouraged to make up this test and the results.
Shadowrun Missions 3-09: Something Completely Different
The Shadowrun mission, Something Completely Different (4th edition) has players encounter the Dweller. Though in that mission, they aren't really on a planar quest and encounter the Dweller under different conditions. They still have to pass a series of test before moving to the next stage of this encounter. There is no actual penalty for failing the tests, other than wasting time and retrying.