The penalty applies when you make an attack with a weapon in your secondary hand, unless you have a way to remove the penalty
Two Weapon Fighting (hereafter TWF) says:
A character must use one of the following in a secondary hand: Buckler, Dagger, Hand Weapon, Main Gauche, Shield, or Sword-breaker.
A character may use either hand to make an attack. This does not give any extra attacks. Attacks from a secondary hand suffer a –20% Weapon Skill penalty.
-- WFRP p 130
And the text in the Balanced weapon property reads as follows:
Weapons with this quality are designed to be used in your secondary hand, usually in concert with a rapier or foil. When wielding a Balanced weapon in your secondary hand, you don’t suffer the normal –20% penalty to your Weapon Skill
-- WFRP p 105
The "normal -20% penalty" it refers to is the penalty mentioned above in TWF.
The TWF rules explicitly mention there is a penalty and lists what you can wield. The advantage of a weapon with the Balanced property (there appear to be 3: Buckler, Main Gauche, and Sword-breaker, as listed in table 5-4 on page 108) is that you don't take the penalty because the weapon you're using was "designed to be used in your secondary hand" in such a way that is doesn't incur the usual penalties. Shields, for example, while listed in TWF as valid options for your secondary hand, do not have the Balanced property likely because they are more unwieldy than a buckler:
[...] Due to the large size of the shield [...]
-- WFRP p 111
This should apply to other hand weapons like swords that weren't "designed to be used in your secondary hand." This doesn't seem to be explicitly stated, but it is implied and seems to make sense within the context of the rules.
Ambidextrous, as you mention, states:
You can use either hand equally well. You do not suffer the normal –20% WS or BS penalty when using a weapon in your secondary hand.
-- WFRP p 97
This again refers back to the rule in TWF.
Dodge and Parry states:
A character can attempt to parry a successful attack, either by using the parrying stance action or by taking advantage of a weapon in his secondary hand.... A Weapon Skill Test is made to parry the blow.... Parrying requires no special skill or talent to attempt. It’s a half action to enter a parrying stance, but if you have a weapon in your secondary hand you can parry as a free action
-- WFRP p 129
Note that the TWF rule specifies "Attacks from a secondary hand" (emphasis mine) incur the penalty. Since Parrying isn't an attack, you don't incur the -20% penalty when you parry.
So, to sum it all up:
When wielding a weapon in your secondary hand, you incur a -20% penalty to attacks made with that weapon, unless that weapon has the Balanced property or you have the Ambidextrous skill.
There is a small conflict in the Ambidextrous rule vs the TWF rule. Ambidextrous says "-20% WS or BS penalty," referring to both Weapon Skill and Ballistic Skill, while TWF says "-20% Weapon Skill penalty." The rules for TWF do not say you can wield a ranged weapon in your secondary hand but it seems implied by the Ambidextrous rule that you can, since the rule wouldn't ever apply to BS unless you were wielding a ranged weapon as a secondary weapon. I'm unsure if this is an error or an oversight, it's just something I noticed.