If you have the Improved Unarmed Strike feat, you do not get the -4 penalty.
When the game rules talk about being unarmed, they talk about not wielding a weapon. Improved Unarmed Strike means that you know how to handle yourself in an unarmed fight and in effect are Armed (are wielding a weapon) because you have a weapon on hand (even if it is literally just your hand) even though you aren't holding weapon. People without the feat are assumed to be unable to effectively handle themselves in a fight without a weapon.
Think of it as the difference between a regular person on the street, and a martial artist, or boxer.
Benefit: You are considered to be armed even when unarmed—you do not
provoke attacks of opportunity when you attack foes while unarmed.
Your unarmed strikes can deal lethal or nonlethal damage, at your
Normal: Without this feat, you are considered unarmed when attacking
with an unarmed strike, and you can deal only nonlethal damage with
such an attack.