What about weapons that have both the Agile and the Trip Trait? There are some, most of them monk/"eastern" weapons, but also the sickle. Would the Athletics check on the second Action get a -4 because of the weapons Agile trait?
Trip benefits from Agile
The Trip trait states:
You can use this weapon to Trip with the Athletics skill even if you don't have a free hand. [...]
And the Trip action has the Attack trait wich states:
An ability with this trait involves an attack. For each attack you make beyond the first on your turn, you take a multiple attack penalty.
The Trip action is an attack, and if you do it via a weapon's trip trait, then the Trip action is an attack made with a weapon.
That matches all the wording of the Agile trait, which states:
The multiple attack penalty you take with this weapon on the second attack on your turn is –4 instead of –5, and –8 instead of –10 on the third and subsequent attacks in the turn.
Trip does not benefit from Agile (or Finesse)
An errata (under Pathfinder Core Rulebook Errata) clarifies that Paizo officially does not support using an Finesse/Agile+(Skill Action) weapon to use Dex or reduce MAP to said skill check "Attacks".
Page 446: Attack Rolls. There was some confusion as to whether skill checks with the attack trait (such as Grapple or Trip) are also attack rolls at the same time. They are not. To make this clear, add this sentence to the beginning of the definition of attack roll "When you use a Strike action or make a spell attack, you attempt a check called an attack roll."
To clarify the different rules elements involved:
An attack is any check that has the attack trait. It applies and increases the multiple attack penalty.
An attack roll is one of the core types of checks in the game (along with saving throws, skill checks, and Perception checks). They are used for Strikes and spell attacks, and traditionally target Armor Class.
Some skill actions have the attack trait, specifically Athletics actions such as Grapple and Trip. You still make a skill check with these skills, not an attack roll.
The multiple attack penalty applies on those skill actions as well. As it says later on in the definition of attack roll "Striking multiple times in a turn has diminishing returns. The multiple attack penalty (detailed on page 446) applies to each attack after the first, whether those attacks are Strikes, special attacks like the Grapple action of the Athletics skill, or spell attack rolls." There is inaccurate language in the Multiple Attack Penalty section implying it applies only to attack rolls that will be receiving errata.
Because of this, weapon traits that apply to "attack rolls" do not apply to Athletics skill uses with the attack trait.
@Ruse's answer is a perfectly valid reason to homerule against this; just make sure to talk it over with your table(s).