tl;dr Multiple Checks
- Fighter makes a Strength check
- Rogue makes a Strength (Athletics) or Dexterity (Acrobatics) check
- Rogue gets advantage on the attack roll
Long version
There are multiple parts to this situation and we need to find a ruling for each of them. Your instincts were good and I wouldn't be too worried about how you handled it at the table. I also agree its a good idea to learn what the rules say so you can make an informed decision should the situation arise again. So without further ado here is how I would rule this.
The fighter with the shield
By a strict reading of the rules this action would be classified as the 'help' action and doing it would consume the fighter's main action. Under rules-as-fun this is a terrible ruling though and effectively punishes the players for trying something cool by harming their action economy. Instead I would rules this uses the fighters reaction. Under reactions in the PHB we have (emphasis mine):
Certain special abilities, spells and situations allow you to take a special action called a reaction.
I believe this situation would qualify as one worthy of a reaction. So what roll do they need to make? Under 'using each ability' we have the following rule for strength:
A Strength check can model any attempt to lift, push, pull, or break something, force your body through a space, or to otherwise apply brute force to a situation.
From that description a strength check would be the appropriate one for the fighter to make. You may choose to make it a Strength (Athletics) check if you like but the rules provide no support other than not ruling it out.
The difficulty class for this check should be based on how heavy the rogue is and how far they are trying to throw them. I would suggest starting with a DC of 10 and increasing for a heavy rogue, or if they are trying to throw them an exceptionally long way, use the rogues normal jump distance as a rough guide. I would add 1 to the DC for each additional foot you attempt to throw them beyond their typical jump distance.
So the fighter uses their reaction to make a DC10 (+ distance) Strength check to throw the rogue with their shield. If they fail but still exceed DC10 I would allow the rogue to make the jump but not allow any additional height/distance from the fighter pushing upwards. If they fail to make DC10 the fighter crumples under the weight and cannot hold the shield steady.
The rogue making the leap
For the rogues jump no action is require as it can be included as part of their movement. From movement and position in the PHB (emphasis mine):
Your movement can include jumping, climbing and swimming. These different modes of movement can be combined with walking, or they can constitute your entire move. However you're moving, you deduct the distance of each part of your move from your speed until it is used up.
So long as the rogue has enough movement they can make the jump without using an action or bonus action. Since they are starting 20 feet away and moving 20 feet up, unless they have increase mobility (fast race, feat or multi-classing) they will need to use the cunning action (bonus action) to dash in order to have sufficient movement.
From the special types of movement rules we have:
High Jump. When you make a high jump, you leap into the air a number of feet equal to 3 + your Strength modifier if you move at least 10 feet on foot immediately before the jump. [...] In some circumstances, your DM might allow you to make a Strength (Athletics) check to jump higher than you normally can.
[...] You can extend your arms half your height above yourself during the jump. Thus, you can reach above you a distance equal to the height of the jump plus 1 1/2 times your height.
Depending on the rogues race and strength modifier this may or may not be high enough to reach the tall enemy. If it is not you may like to make them roll a Strength (Athletics) check with a DC equal to 10 + the number of additional feat required (my house rule). You may like to allow the fighter's strength check to replace this check provided they succeed, or allow the two to combine both results to reach even greater heights.
Acrobatics. Your Dexterity (Acrobatics) check covers your attempt to stay on your feet in a tricky situation, such as when you're trying to run across a sheet of ice [...] The DM might also call for a Dexterity (Acrobatics) check to see if you can perform acrobatic stunts, including dives, rolls, somersaults, and flips.
The acrobatics rules do not cover jumping which is handled by Strength (Athletics) in the rules. However if the rogue is not jumping but being thrown by the fighter you could rules that keeping their feat on the shield or positioning themselves in the air to make the attack requires a Dexterity (Acrobatics) check. I would suggest DC 10 for this check.
So the rogue makes either a Strength (Athletics) or Dexterity (Acrobatics) check to make the leap during their movement. Don't make them roll both, one check should be sufficient. If they don't have enough speed to cover the entire leap they can use their bonus action to dash with cunning action.
The attack roll
The rules have no support for this kind of attack and have no specific ruling on them. The is no rule on aiming at specific body parts or on special attacks other than the general case from the advantage and disadvantage rules.
The DM can also decide that circumstances influence a roll in one direction or the other and grant advantage or impose disadvantage as a result.
This DM (me) would definitely decide that these circumstances influence the attack roll positively and grant advantage on the attack. In doing so it grants the rogue sneak attack even if the fighter is not within 5 feet of the target.