You add three dice to your original pool, not to the chance die.
From the section on Willpower, p. 132:
Willpower should be spent and three dice are added to your dice pool before all penalties for your roll have been applied. So, form your pool, add all bonuses — including your three dice from Willpower — and then subtract all penalties. Yes, that means if penalties are sufficiently extreme, you could be reduced to a chance roll even if you have spent Willpower for three extra dice.
In fact, the example in the book involves just this situation and concludes, "You must make a chance roll for your character to succeed, even though you spent Willpower on the task."