Here's a function which implements this mechanic:
function: edgesnag EDGE:s SNAG:s {
loop X over {1..#EDGE} {
if X@EDGE > X@SNAG { result: X@EDGE }
if X@EDGE < X@SNAG { result: #SNAG@SNAG }
}
result: [lowest of #EDGE@EDGE and #SNAG@SNAG]
}
As in your previous question, we've defined a function edgesnag which expects to be given two dice pools for edge dice and snag dice and will cast those to sequences (with :s) to fix them for inspection inside the function. Anydice sorts the rolled pools into descending order by default.
We then loop over the pools to inspect the dice, starting with the first and therefore highest die in each sequence. If the edge die is greater than the snag die, that's our result; we return that edge die's value. If the edge die is lower than the snag die, we've been "snagged" (for lack of a better term) and our result is the lowest snag die. If neither of those conditions were true, the dice must be tied and so we loop on to the next pair.
Note that if we had more edge dice than snag dice, we might end up comparing an edge die to a snag die which doesn't exist. In anydice, if you try and access a nonexistent value in a sequence - for instance, 4@{3,2,1} - the value is 0. Since the value of the nonexistent snag die would be 0, the edge die will win this comparison and that'd be our value.
The loop ends when we have compared the last edge die - assuming we didn't find a pair where the edge die won and we didn't get snagged either. In this case, all the edge dice have been cancelled out so we take the lowest roll available - found in this case by comparing the last die of each sequence using the [lowest of X and Y] built in function.
Here's an anydice program which uses this function and some example outputs (including using your given examples to demonstrate it gives the correct result). As it happens it's more efficient to compute this version of your mechanic than the last one so we're able to try it with slightly larger dice pools than last time!
This version of your mechanic doesn't seem to have any counter-intuitive points where getting more snag dice is better or getting more edge dice is worse - unless it's possible to have zero snag dice, in which case adding a single snag die increases the expected result (since it's the only snag die, if it is higher than any of the edge die, it gets chosen as the result, which is better than the edge dice rolled!) However if you assume a minimum of 1 snag die on all rolls then this isn't a problem, adding more snag dice always makes the expected result worse.
