Typically any situation which favors one and disfavors another is already covered by rules in the PHB or flat out ruled impossible.
For instance the case of size you listed above is already factored into grapple checks:
The target of your grapple must be no more than one size larger than
you, and must be within your reach.
In the context of the grapple, size is already considered as trying to grapple something that is more than one size larger is flat out deemed impossible. While you are always allowed to try to escape from the grapple at no penalty.
However, assuming that context was not there, the game rules imply that the advantage or disadvantage only happen once for a specific favorable condition - at the discretion of the DM of course. So if you were going to impose adv or disadv based on size, you should not apply advantage to the check made by the grappler and also impose disadvantage to the person trying to avoid it without having a different reason while the person avoiding it should receive disadvantage. (For instance, the grappler is larger than the grappled target and the grapple target just happens to be blinded by a spell).
PHB 173 If multiple situations affect a roll and each one grants
advantage or imposes disadvantage on it, you don't roll more than one
additional d20. If two favorable siutations grant advantage, for
example you still only roll one additional d20.
These rules don't explicitly prevent the DM from doing what your suggesting, but consider the conflicts you can create if you assign advantage and disadvantage this way. Use the following example to see the potential problems:
I could pose advantage on the grappler initater's contest check because he's large. The opposed check for the contest is either Strength (Athletics) or Dexterity(Acrobatics). Thus I could rule that because of your size difference your Strength isn't likely to help you so your Strength(Athletics) roll if used suffers disadvantage as well but your Dexterity isn't directly affected by my opponents size so I may rule my Dexterity(Athletics) is unaffected. Lets say you do cover yourself in something slippery, you could receive Disadvantage to your Strength(Athletics) if used to defend against the grapple but receive Advantage to your Dexterity(Acrobatics) used to prevent a grapple.
As you can see this does serve inhibit the defending player's ability to use Strength while still making the grappler have a strong roll regardless of which roll the contested opponent uses.
However, you will notice the contradiction in rationalization that I just made in the roleplay. I just tried to impose both advantage to the attacker and disadvantage to the defender -okay good so far. However, as the defender, I ended up using my dexterity which I ruled wasn't affected by size, but then I gave advantage to the grapple iniater's check because of size. So I essentially imposed a penalty to the grappled persons role by ruling the grappler gets advantage, and my reason for imposing the penalty is no longer valid and rational.
This is why the notion of advantage and disadvantage is typically unidirectional. In the example it would have been more appropriate to leave the grapple initiater's check alone but impose disadvantage on Strength check of the grapplee. It is not pointless to do this at it inhibits the defending character's ability to use Strength to counter the grapple, and capture's your point that size is a problem. In this context however, the attempt to avoid the grapple is not "impossible" if I attempt to use my Dexterity.