I've written the following anydice function which I think implements your desired mechanic, as described. I have assumed that, when cancelling, the highest snag die cancels the highest edge die that it can, and then so on descending from there.
function: edgesnag EDGE:s SNAG:s {
loop X over {1..#EDGE} {
if X@EDGE >= X@SNAG { result: X@EDGE }
}
if #EDGE < #SNAG { result: #SNAG@SNAG }
result: 1
}
This function expects to be invoked with two dice pools, e.g. [edgesnag 2d6 3d6].
So the way this works is that we cast the die pools to sequences, to fix them for inspection, and then we'll loop over those sequences to compare the dice. Remember that by default therefore anydice sorts the pool in descending order (so a roll like 3, 5, 3 becomes {5,3,3}). We thus consider the dice in the two sequences in pairs.
If the first die in the EDGE sequence is greater than or equal to the first die in the SNAG sequence, we are already certain that this edge die will not be cancelled by any snag dice, so it will remain at the end and is therefore the result. Otherwise, the snag die cancels the edge die, and we loop on to the next pair, comparing them in the same way.
If we run out of snag dice to cancel edge dice without finding a keeper, we'll take the edge die we're currently considering as the highest remaining result (the value of a nonexistent element in a sequence is 0, so the check will favour the edge die). Otherwise, the loop will continue until we run out of edge dice.
If we run out of edge dice, and there were more snag dice than edge dice, snag dice must remain and so the result will be the smallest snag die. Otherwise, it must be the case that no dice remain, and the result is 1.
It is necessary for us to optimise the algorithmic implementation of your mechanic in this way because otherwise, anydice will choke pretty quickly on relatively small dice pools. Even as it is, it can just about handle [edgesnag 5d6 5d6]; beyond this, the possible sequence permutations are too large and it chokes. (Some greater anydice expert might be able to further optimise it past this limitation but this is as far as I go.)
Observations
A couple of observations about your mechanic.
Firstly, because snag die only cancel lower edge die, an edge die of 6 will never be cancelled, this skews your results very heavily towards sixes - no matter how large the snag dice pool is, it doesn't change the odds of getting a result of 6, only alters the rest of the distribution.
You also get some funky crossover points where your average result improves when you have more snag die, because you can only ever end up taking the lowest snag die when snag dice outnumber edge dice, and the lowest snag die might be greater than 1. For instance, if your edge pool is 1d6, and your snag pool is 1d6, your average result is 2.94; but if your snag pool is 2d6, you average 3.66, or 3.17 for 3d6. (anydice link)
