WARNING: ONLY A (good) APPROXIMATION!
.
Number chances
After fiddling around with anydice for several hours and going through a fully false result, I finally got the following code. It tells how often each number occurs on the die combination:
DONE:d6
NONE:2
DTWO:d8
NTWO:3
function: run{
result: (NONE d (DONE =NUMBER)+ NTWO d (DTWO=NUMBER))
}
loop NUMBER over {1..[maximum of DTWO]}{
output [run] named "[NUMBER]s on [NONE][DONE]+[NTWO][DTWO]"
}
Counting out
However, that is not yet counted out and pretty user unfriendly. So more fiddling resulted in this:
DONE:d6
NONE:2
DTWO:d8
NTWO:3
function: run{
COUNTER:0
loop RUNS over {1..AMMOUNT}{
COUNTER: COUNTER+ (NONE d (DONE =NUMBER)+ NTWO d (DTWO=NUMBER))=RUNS
}
result: COUNTER
}
loop AMMOUNT over {1..2}{
loop NUMBER over {1..[maximum of DTWO]}{
output [run] named "[AMMOUNT] times [NUMBER] on [NONE][DONE]+[NTWO][DTWO]"
}}
This at least tells us how many of those cases are 1 or 2 of those. We don't have an interest in cases where 3 or more of a given result pop up, as those extras will have to be ignored. Still, that only gives us the plain "once or twice" and "twice the same" probabilities, not the probabilities for the highest two of the set.
Now, working that out is clearly not done yet..
Sequencing
But here sequences come in handy: Anydice can make sequences, so we can make a sequence of NdX and MdY, that is a single string holding all those numbers. It handles almost like a die, but isn't one.
DONE:d6
NONE:2
DTWO:d8
NTWO:3
output {DONE:NONE,DTWO:NTWO} named "1,2,3,4,5,6,1,2,3,4,5,6,1,2,3,4,5,6,7,8,1,2,3,4,5,6,7,8,1,2,3,4,5,6,7,8"
Getting Sequenced Solutions
This sequence can't be used as a die as it is, we must turn it into one. To do this, we pretend it is an N-sided super-die (where N length of the sequence) which is labeled in the same manner as the sequence, so in our example 5x1, 5x2, ..., 5x6, 3x7, 3x8. The easy part is, we just need to add a "d" in front of the sequence and got a die.
To get the final result of highest two out of one roll set, we just need to roll a number of these imaginary super-dices equal to the total number of real dice we plunged into the super-dice involved.
DONE:d6
NONE:2
DTWO:d8
NTWO:3
output [highest 2 of (NONE+NTWO)d{DONE:NONE,DTWO:NTWO}]
For the test case NONE:0, with any NTWO, it does hold the correct result, so here we are, closing in on the Home Stretch! This still has some errors: for NONE:1 NTOW:1 the code allows 15 and 16, which shouldn't be. That case is actually the highest of 1d6 and 1d8, always taking both. We can't have 2d6 and 1d8 as of the premise. For 2d6 we fall back to the simple case of keeping both, so we need a to check some cases separately to fix the results.
Case analysis
Case differentiation for NTWO=0, 1 and more does fix the faulty results in the low regions:
DONE:d6
NONE:2
DTWO:d8
NTWO:0
function: run{
if (NTWO>1){result: [highest 2 of (NONE+NTWO)d{DONE:NONE,DTWO:NTWO}]}
if (NTWO=1){result: [highest 1 of d6]+[highest 1 of d8]}
if (NTWO=0){result: [highest 2 of 2d6]}
}
output [run]
Final Words
The solution gives at best a very good approximation but, as Ilmari pointed out:
[Y]ou're effectively defining a hybrid die that behaves like a d6 with probability N1/(N1+N2) and like a d8 with probability N2/(N1+N2), and then making N1+N2 independent rolls of that die. Thus, while the dice pool you're rolling will have an average of N1 d6's and N2 d8's, you also end up counting cases where the composition of the pool differs from the average