Helping is straightforward in most circumstances, but it can get complicated
There are two primary ways that you can assist someone in D&D 5e:
- Helping them and thereby giving them advantage on the roll (which is the equivalent of a +5 modifier on the roll in general)
- Participating instead in a group check (for two participants this equates to "each of you roll and if one of you succeeds then you both succeed...which is strictly worse than the advantage mechanic for two characters)
Both of these options are pretty straight-forward.
Granting them advantage on their roll
The rules on working together state:
Sometimes two or more characters team up to attempt a task. The character who's leading the effort — or the one with the highest ability modifier — can make an ability check with advantage, reflecting the help provided by the other characters. In combat, this requires the Help action (see chapter 9, “Combat”).
A character can only provide help if the task is one that he or she could attempt alone. [...] Moreover, a character can help only when two or more individuals working together would actually be productive.
So, the anatomy of this is:
- By receiving help, the character with the highest ability modifier (or the character who is leading the check) gets advantage on the roll
- You can't get double advantage, so only receiving help from one other character is really useful
Participating in a group check
The section continues by presenting the rules on group checks:
When a number of individuals are trying to accomplish something as a group, the DM might ask for a group ability check. In such a situation, the characters who are skilled at a particular task help cover those who aren't.
To make a group ability check, everyone in the group makes the ability check. If at least half the group succeeds, the whole group succeeds. Otherwise, the group fails.
The only question here is what happens if we have an odd number of characters in the group?
Well here, one of the general rules listed at the start of the PHB applies: Round down.
There’s one more general rule you need to know at the outset. Whenever you divide a number in the game, round down if you end up with a fraction, even if the fraction is one-half or greater.
So, a group check with three participants only requires one of the three participants to succeed for the group to succeed. If you have four participants it requires two to succeed, and so on.
But, for your specific situation there is more to the story than just the straight helping rules.
In your specific example, the general rules on the Strength ability would also apply, in particular the lifting and carrying rules:
Your Strength score determines the amount of weight you can bear. The following terms define what you can lift or carry.
Carrying Capacity. Your carrying capacity is your Strength score multiplied by 15. This is the weight (in pounds) that you can carry, which is high enough that most characters don't usually have to worry about it.
Push, Drag, or Lift. You can push, drag, or lift a weight in pounds up to twice your carrying capacity (or 30 times your Strength score). While pushing or dragging weight in excess of your carrying capacity, your speed drops to 5 feet.
Size and Strength. Larger creatures can bear more weight, whereas Tiny creatures can carry less. For each size category above Medium, double the creature's carrying capacity and the amount it can push, drag, or lift. For a Tiny creature, halve these weights.
In the situation described your two PC characters are attempting to drag the flying creature down to prevent it getting away, while the flying creature is attempting to lift your characters (i.e. continue flying). How much a creature can drag/lift is determined by it's size and strength score.
But is there a precedent for using the rules in this way?
Yes, in a number of the adventures published by WotC (Tomb of Annihilation, Out of the Abyss, Waterdeep: Dungeon of the Mad Mage, Divine Contention, Tales from the Yawning Portal to name but a few), there is text to the effect:
Characters with a combined Strength score of XX or more can lift/push/move/topple/drag [insert object].
This mirrors what we see in our own world, where one person cannot move a heavy thing, but many people can easily move a thing with their combined strength.
How do we resolve this?
The game has a general rule for whether or not we even get to roll dice to make an ability check:
The DM calls for an ability check when a character or monster attempts an action (other than an attack) that has a chance of failure. When the outcome is uncertain, the dice determine the results.
The DMG (p. 237) also provides some guidance on how to adjudicate in these sorts of situations:
When deciding whether to use a roll, ask yourself two questions:
- Is a task so easy and so free of conflict and stress that there should be no chance of failure?
- Is a task so inappropriate or impossible — such as hitting the moon with an arrow — that it can’t work?
If the answer to both of these questions is no, some kind of roll is appropriate.
So, in order to decide if a roll is required we need to determine if the task being attempted is possible (for both sides). To do that we need to compare the total drag capacity of the PCs (or their combined weight, whichever is higher) against the total lift capacity of the flying creature.
The DM might also decide to use the Variant: Encumbrance rules presented in the Strength section to determine if the creature can fly with all of the weight it is trying to lift:
If you carry weight in excess of 5 times your Strength score, you are encumbered, which means your speed drops by 10 feet.
If you carry weight in excess of 10 times your Strength score, up to your maximum carrying capacity, you are instead heavily encumbered, which means your speed drops by 20 feet and you have disadvantage on ability checks, attack rolls, and saving throws that use Strength, Dexterity, or Constitution.
So we have two options:
- One side of the contest clearly dominates and it is impossible for the other to even attempt to succeed
- It's close, and a roll is required
One side domination...
If one side clearly outmatches the other then no die roll is required for the opposed strength check, the contest is clearly won. For example if your two PCs are attempting to drag down a Roc by grabbing onto it's claws they will fail, but if a grappled flying creature is a bat it will clearly fail.
This, of course, does not preclude the grappled creature from attempting to escape the grapple:
Escaping a Grapple. A grappled creature can use its action to escape. To do so, it must succeed on a Strength (Athletics) or Dexterity (Acrobatics) check contested by your Strength (Athletics) check.
It is possible for a creature to use the Help action to grant the grappled creature advantage on their roll to escape the grapple. It's even possible, if the DM is using the Encumbrance rules, that the flying creature has disadvantage on it's strength check to try maintain the grapple.
If instead the two totals are close (for some definition of close determined by the DM) then a Strength or Athletics contest is appropriate. 5e has rules for ability check contests:
Sometimes one character's or monster's efforts are directly opposed to another's.
This situation also applies when one of them is trying to prevent the other one from accomplishing a goal [...]. In situations like these, the outcome is determined by a special form of ability check, called a contest.
Both participants in a contest make ability checks appropriate to their efforts. They apply all appropriate bonuses and penalties, but instead of comparing the total to a DC, they compare the totals of their two checks. The participant with the higher check total wins the contest. That character or monster either succeeds at the action or prevents the other one from succeeding.
If the contest results in a tie, the situation remains the same as it was before the contest. [...]
In this case, your PCs are attempting to prevent the creature from flying away. Combining everything we have found the result is:
- One PC gets to make a Strength (Athletics) check with advantage
- The creature gets to make a straight Strength (Athletics) check
- The PCs must beat the creatures result or the creature will continue to fly away
To resolve the situation:
- Check if a roll is even required (ie is it possible for the PCs to restrain the creature as described)
- If it isn't required because one side wins, the creature can still attempt to escape the grapple
- If a roll is needed, then the rules for contested checks come into play, and one of the PCs will have advantage on the roll
- The PC can still attempt to escape the grapple (possibly helped by their ally), and if they are so helped will have advantage on the roll