That dice mechanic – pools of d10s, high numbers matter but so do matches and sometimes runs – sounds like the One Roll Engine.
ORE is a little different than you describe, but pretty close: you don't look for single high numbers, but sets of matched numbers in pairs or more, and both the face number of the set (called "height") and the number of matching dice (called "width") matter to what happens. (There are also ORE variants that use runs of numbers like you describe.) Usually height determines the quality of the result, while width determines how skillfully it was done. For example, when hitting someone, height determines hit location with higher hit locations being more vulnerable (10 is the head), while width determines the damage. Hence the "one roll" in the system's name.
The One Roll Engine is used in a lot of games, but the only straight-up fantasy game that uses it is Reign (if you want a system with a setting) or Reign Enchiridion (if you just want the system from Reign). If you want to take a look at a ORE game to see if the mechanic matches what you're looking for, Nemesis is a free ORE game you can download. If it does turn out to be the system you heard of and you want to buy a fantasy game using it, the Reign Enchiridion is a dense book for a mere $10, and that gets you print and PDF.