For the particular kind of pattern you describe ("number of dice showing one of a particular set of faces"), the most efficient solution is to start by defining a custom die with its faces relabeled as either 1 (for the faces we want to count) or 0 (for the faces we don't want).
Since comparisons between dice and numbers in AnyDice evaluate to 1 if true and to 0 if false, this can be done simply by comparing the die to a target number and assigning its result to a variable, e.g. like this:
X: d6 >= 5
or (equivalently):
X: [count {5, 6} in d6]
Once you have the custom die, you can roll Y of them and count the number of matching faces simply with:
output YdX named "number of 5s and 6s on [Y]d6"
As for rerolls, it turns out the we can apply a simple trick: since we know that the optimal strategy is to never reroll dice that rolled a matching face (i.e. a face we relabeled with 1) and to always reroll as many dice that rolled a non-matching face (i.e. a face we relabeled with 0) as possible, we can instead just roll Z extra dice (where Z is that maximum allowed number of rerolls) and take the Y highest of them, like this:
output [highest Y of (Y+Z)dX] named "number of 5s and 6s on [Y]d6 with [Z] single die rerolls"
Here's a full example program demonstrating this: https://anydice.com/program/2f089
Update: I assumed above that by "Y dice with Z rerolls" you meant being able to reroll one of the Y dice (not necessarily always the same one) up to Z times.
If you instead meant being able to reroll all of the Y dice at once up to Z times, until you get a good enough roll, you can do that efficiently like this:
output [highest 1 of (1+Z)d(YdX)] named "number of 5s and 6s on [Y]d6 with [Z] full rerolls"
Now, that code really needs a disclaimer. What it's really calculating is the distribution you'd get by rolling 1+Z sets of Y dice and picking the best one of those sets. That's of course not quite the same as being able to reroll a single set of Y dice up to Z times, since you won't know what your future rolls will be when you need to decide whether to reroll.
But if you're just trying to roll at least a certain result, then the optimal rerolling strategy is simple: always reroll if you're below the target, and never reroll if you're at or above it. And it's not hard to see that the only way to fail to hit the target this way is to use up all of your Z rerolls (and your single original roll) and fail all of them. So your chance of rolling at least your target number with this strategy is the same as if you made all 1+Z rolls in advance and just picked the best one.
(Of course, the best roll might be even better than the first successful one, but a success is a success either way.)
So the correct way (for your mechanic) to read the results from this program is to look at it in "At Least" mode:
…and interpret e.g. the 53.29% value for a target number of 4 as the probability of getting at least four 5s and 6s on 7d6 if you can reroll all the dice up to three times and will stop as soon as you get at least four 5s and 6s. And similarly the 16.91% is the probability of getting at least five 5s and 6s if you stop rerolling as soon as you do. But if you stop at four 5s and 6s, you will not get five of them with 16.91% probability.
Update 2: As noted by HighDiceRoller, there's a third possible interpretation: you might mean that you can reroll any subset of the Y dice up to Z times.
Now, for your specific type of pattern (where you just want to roll as many of a particular set of faces as possible, and whether you want to keep or reroll a particular die doesn't depend on any of the other dice) this is equivalent to being able to reroll each individual die separately up to Z times. And thus, we can again model it efficiently in AnyDice just by defining a "best of 1+Z" custom die (based on our relabeled die X
above) and then rolling Y of them:
Q: [highest 1 of (1+Z)dX]
output YdQ named "number of 5s and 6s on [Y]d6 with [Z] rerolls of any subset"
Here's a program implementing all three of these possible reroll mechanics: https://anydice.com/program/2f116
(BTW, this is a much stronger reroll mechanic than either of the previous two, effectively combining both of their strongest features: you get just as many potential rerolls per die as with the "reroll all Y dice" mechanic, but you also get to keep all of your "good" rolls instead of having to reroll them along with the bad ones.)