Weapon Size states:
The measure of how much effort it takes to use a weapon (whether the weapon is designated as a light, one-handed, or two-handed weapon for a particular wielder) is altered by one step for each size category of difference between the wielder's size and the size of the creature for which the weapon was designed.
That means that a Small Whip is considered a light weapon for a medium sized character (1 step of size difference)
In this case, you could use Two-Weapon Fighting with a -2/-2 penalty, plus the -2 for attacks with an inappropriately sized weapon, making it -2/-4, as you thought.
This topic on the paizo messageboards also addresses this question.
Effect of Weapon and Creature Sizes on Reach
Nothing in the rules about weapon special qualities (such as reach) or Weapon Size says anything about weapon special qualities not applying for inappropriately sized weapons. This means that the reach quality is not altered at all by weapon size. If you were somehow able to handle a Gargantuan Whip (while remaining less than large yourself), it would still have 15ft. reach.
Reach Weapons states:
[...] Most reach weapons double the wielder's natural reach [...]
to a minimum of 5ft. reach for tiny or smaller creatures (see linked post below).
While this rule mostly works, the whip is different. According to the text, it just grants 15 ft. reach, instead of saying the reach is tripled. There is no text concerning the effect of creature size here.
The most official ruling on this is this post by James Jacobs, Creative Director at Paizo. The whip triples the creatures natural reach indeed, to a minimum of 10ft. for tiny or smaller creatures.