How can I calculate my chance of success given my
ability + skill vs a
difficulty? How is this complicated when the difficulty is rolled as well?
I would recommend using anydice.com. The creator of anydice.com has written a program on that site to calculate success probabilities for NWoD, but the dice mechanics are very similar and shouldn't need much tweaking. Here is the post. It should give you a rough idea.
I tweeted the creater of Anydice.com and he suggested this:
Which gives this sort of graph:
This is a question that many WoD players have faced and tackled. Here is a good start.
Difficulty shouldn't be rolled, but set at a number (such as 6) and opposing rolls compare successes to determine victor.
A fast and dirty (and not terribly precise) method I recommend for my players of both Shadowrun and V20 is as follows:
Take what percentage of the dice is less than the target number (I.e. 4 on a six-sided die would have a 50% success chance, 6 on a 10-sided die has 50% chance).
For every die past that add half of the chance of the previous die; 1 d6 versus 4 is 50%, 2 is 75%, 3 is 87.5%, so on and so forth, and the same progress applies to d10 on oWoD dice. I'm not familiar with nWoD's system, so I can't help, but I'm assuming if you're playing Masquerade you're using the old system. This doesn't apply so much to V20 since I play with fumbles, so there needs to be a calculation (10% for each die past the first reduced similarly to the normal success chance seems a good rule). This allows them to quickly guess more complex chances, like if the target number were increased (40%; 50%; 55%). It breaks down rather quickly as you throw more dice, however, but usually isn't such a big deal, though it does work a lot better for Shadowrun than it does for V20.