This is not an AnyDice-based solution, but is otherwise directly responsive. I made a dyce-based¹ interactive notebook that provides extended visualization options beyond those AnyDice supports and that you can use to tinker with various inputs to your mechanic. You can play around with it in your browser: 
[source]
While a matter of taste, I find anydyce's² "burst" graphs (above) often help give insight into the feel for distributions like these. But distribution is a small part of mechanic "feel", on which I'll elaborate and opine further below. For now, though, the core mechanic implementation used to create the above graphs is as follows:
class ResultType(IntEnum):
IMPOSSIBLE = auto() # no roll can succeed because target <= challenge die
TARGET_MISS = auto() # failure because player die >= target
CHALLENGE_MISS = auto() # failure because player die <= challenge die
HIT = auto() # success
def expected_result(
player_die: H,
challenge_die: H,
target: int,
) -> H:
def _dependent_term(player_die: HResult, challenge_die: HResult):
if target <= challenge_die.outcome:
return ResultType.IMPOSSIBLE
elif player_die.outcome >= target:
return ResultType.TARGET_MISS
elif player_die.outcome <= challenge_die.outcome:
return ResultType.CHALLENGE_MISS
else:
return ResultType.HIT
# Start with zero counts for all possible outcomes
result_base = H((outcome, 0) for outcome in ResultType)
# Accumulate those that came up in our calculation
return result_base.accumulate(foreach(_dependent_term, player_die, challenge_die))
If I'm reading your original description accurately, one has to roll less than (not less-than-or-equal-to) the target number and greater than (not greater-than-or-equal-to) the challenge die outcome. If I've misread, then the above code (and it's place in the linked interactive notebook) can be modified accordingly (changing >= to > and <= to <).
If you don't want the fidelity of whether the die failed because of the target or because the challenge die, there's a simpler approach akin to Dale M's answer:
def expected_result_low_fidelity(
player_die: H,
challenge_die: H,
target: int,
) -> H:
# Build a die that has just the faces that are below the target
# (or zero where they were at or above)
player_target_die = H(
(outcome, count) if outcome < target else (0, count)
for outcome, count in player_die.items()
)
# Then compute a histogram for where those faces are greater than
# the challenge die
return player_target_die.gt(challenge_die)
That being said, I think understanding where/when failures can occur provides important insight into how the mechanic might feel at the table, akin to how the popular Advantage mechanic feels a lot different than merely adding or subtracting a statistically equivalent static modifier:
What matters is how Advantage and Disadvantage feel at the table. Rolling two dice and taking the higher — or being forced to take the lower — feels way better — or way worse — than rolling one die and adding — or subtracting — an extra number. Moreover, the play dynamic creates all these interesting little emotional outcomes. If you have Advantage and roll two ones, that’s a f$&%ing gut punch, for example. And if you roll a one and a seventeen, it feels like Advantage saved your a$& from disaster, even if, statistically, there’s no way to know whether you’d have gotten the one or the seventeen if you’d just rolled a single die.
"The Best and Worst of D&D 5E" (Rehm, Dec 2022)
Revealing the target and challenge roll to the player or changing the order rolls are made could elicit significantly different reactions. Depending on parameters, there could be situations where no roll the player makes would succeed. Consider rolling the challenge die first, which would reveal those helpless situations without player action. Consider instead naming the target, then having the player roll a die, then rolling the challenge die (or maybe even having the player roll the challenge die) if the player rolls under the target. The player could be relieved by beating the target after rolling a two, only to realize the implications that the challenge die is very likely to change the tide for the worse.
Note that neither implementation above has an opinion about the player die (other than it having exclusively positive outcomes), so you can experiment with that, too (e.g., if you wanted to see how 2d10 or 3d6 worked in lieu of a d20).
¹ dyce is my Python dice probability library.
² anydyce is my visualization layer for dyce meant as a rough stand-in for AnyDice.
