As minnmass notes, AnyDice is a useful tool for these kinds of calculations. Here's a quick AnyDice program that should get you started:
ABILITY: 4
DIE: d6 <= ABILITY
DICE: 3dDIE
function: roll ROLL:s until X:n successes or Y:n failures {
S: ROLL = 1 \ count successes in roll \
F: ROLL = 0 \ count failures in roll \
if S >= X | F >= Y {
result: S
} else {
result: S + [roll DICE until X-S successes or Y-F failures]
}
}
set "maximum function depth" to 99
output [roll DICE until 30 successes or 3 failures] named "successes"
What we're doing is first defining a custom relabeled six-sided die named DIE that rolls a 1 on a success and 0 on a failure, and then a pool of three such dice named DICE. (This is mainly a performance optimization: the fewer differently numbered sides your dice have, the faster AnyDice runs.)
Then we define a function that takes a sequence parameter named ROLL (representing the result of rolling a bunch of the custom dice defined above) and two numeric parameters named X and Y (indicating when we want or must stop rolling more dice). The function counts the number of successes and failures in ROLL and then either just returns the success count (if we hit one of the thresholds and stop rolling) or calls itself (to model a subsequent roll) with adjusted thresholds (subtracting the number of successes and failures already rolled) and adds the result to the success count.
And finally we have an output statement that calls the function and outputs the result. (But before that we need to adjust AnyDice's setting a bit to stop it from complaining about too many nested function calls.)
There's a bit of AnyDice magic going on in the function calls, where we pass DICE in as the value of the ROLL parameter. Remember, DICE is a collection of (unrolled) dice, while the :s in the function definition indicates that ROLL is supposed to be a sequence of numbers. When AnyDice notices this, it does something special: it calls the function for every possible (sorted) outcome of rolling the dice and collects the results of each of these function calls into a custom die weighed by their probability. So, whereas outside the function DICE is just a pool of three dice with no fixed numeric value, inside the function ROLL is a fixed sequence of three numbers that we can examine and manipulate any way we want.
FWIW, this is what the output of the code above looks like in Graph mode:
That's an interesting looking graph! I wonder where those peaks at multiples of three successes come from. Let's see what happens if I adjust the maximum allowed number of failures while keeping the pool size fixed:
And here's what happens if I adjust the pool size instead:
It looks like the spacing between the peaks is determined by the pool size, but the position of the peaks is also affected by the maximum number of failures — apparently allowing one more failure moves the peaks one success to the left.
BTW, note that we're still only outputting the total number of successes rolled. We can tweak the function to also return the total number of failures, but that requires a bit of trickery, since AnyDice can normally only output numbers. However, we can encode multiple numbers into one number, e.g. by mapping the numbers S and F into 100 × S + F, like this:
ABILITY: 4
DIE: d6 <= ABILITY
DICE: 3dDIE
function: roll ROLL:s until X:n successes or Y:n failures {
S: ROLL = 1 \ count successes in roll \
F: ROLL = 0 \ count failures in roll \
ENCODED: 100 * S + F
if S >= X | F >= Y {
result: ENCODED
} else {
result: ENCODED + [roll DICE until X-S successes or Y-F failures]
}
}
set "maximum function depth" to 99
output [roll DICE until 30 successes or 3 failures] named "100 * successes + failures"
As long as F can never be greater than 99, this encoding is unambiguous and easy to read: the last two digits of the encoded number correspond to F, and the digits to the left of them to S. So, for example, the output 603 corresponds to 6 successes and 3 failures.
Alas, the one feature we lose is the ability to (usefully) graph the output. But we can still look at the output in Table mode:
In particular, looking at this output and playing with different maximum failure counts and dice pool sizes, we can see what's going on with the peaks: the total sum of failures and successes must, of course, be a multiple of the pool size, and the peaks simply correspond to the cases where we roll the minimum number of failures needed to stop, as that's more likely than overshooting.
OK, that's interesting, but what about the other things you wanted to investigate?
The code above already lets you easily adjust the ability score, the number of dice rolled at a time, the type of dice used and the limits on how many failures you want to allow and/or how many successes you consider to be enough. That takes care of most of your questions, except for crits and rerolls.
Crit fails on 6s can be handled by relabeling the dice a bit differently: instead of just 1 for success and 0 for failure, we can also add e.g. -1 for critical failure. Probably the most elegant way to construct such dice is with a helper function:
function: relabel SIDE:n {
if SIDE <= ABILITY { result: 1 } \ success \
else if SIDE < 6 { result: 0 } \ normal failure \
else { result: -1 } \ critical failure \
}
DIE: [relabel d6]
We'll also need to adjust our main function to deal with crits, of course, e.g. like this:
function: roll ROLL:s until X:n successes or Y:n failures or Z:n crit fails {
S: ROLL = 1 \ count successes in roll \
F: ROLL <= 0 \ count failures (including crits) in roll \
C: ROLL = -1 \ count critical failures in roll \
if S >= X | F >= Y | C >= Z {
result: S
} else {
result: S + [roll DICE until X-S successes or Y-F failures or Z-C crit fails]
}
}
set "maximum function depth" to 99
output [roll DICE until 30 successes or 3 failures or 2 crit fails] named "successes"
Of course, we can also still use the encoding trick to show the failure and crit failure counts as well:
function: roll ROLL:s until X:n successes or Y:n failures or Z:n crit fails {
S: ROLL = 1 \ count successes in roll \
F: ROLL <= 0 \ count failures (including crits) in roll \
C: ROLL = -1 \ count critical failures in roll \
ENCODED: 100 * S + 10 * F + C
if S >= X | F >= Y | C >= Z {
result: ENCODED
} else {
result: ENCODED + [roll DICE until X-S successes or Y-F failures or Z-C crit fails]
}
}
set "maximum function depth" to 99
output [roll DICE until 30 successes or 3 failures or 2 crit fails]
named "100 * successes + 10 * failures + crit fails"
I'll leave dealing with rerolls as an exercise for now. It should certainly be possible, but incorporating it into the code above feels non-trivial (unless I'm just missing some simple and obvious mathematical trick, which is quite possible).
