They make one attack collectively, so no, their exact number doesn't matter.
Although note that the Greater Spirit Totem rage power causes the wisps to do 1d8 (with no bonuses, but also no need to make an attack roll or save) negative energy damage to every enemy adjacent to the barbarian, and still make their "wisp attack" against any one foe within 15 feet, which could be one who already took the automatic damage or not. In addition to granting the barbarian concealment against enemies any further away than adjacent... if you have a Cha-based barbarian for whatever reason (rage mage? tribal leader? noble scion in need of anger management?), the powers look pretty good.