The fighter's Driving Attack says:
You drive back your adversary with a hail of blows.
Invigorating, Martial, Weapon
Standard Action Melee weapon
Target: One creature
Primary Attack: Strength vs. AC
Hit: 2[W] + Strength modifier damage, and you push the target 1 square. You then shift 1 square to a square the target vacated. Make a secondary attack against the target.
Secondary Attack: Strength vs. Fortitude
Hit: 1[W] + Strength modifier damage, and you push the target 2 squares and knock it prone. Miss: Half damage, and you push the target 1 square.
Me and my adversary are standing in difficult terrain and if I push the target, I won't be able to shift into the vacated square. Therefore, I will choose not to push the adversary. Therefore, I won't be able to shift into a vacated square. Will I be able to make use of the secondary attack anyway?