I have a concept that I want to use of a dryad soothing a demon, which shall be a demon's past as their first step to redemption. So is there any way a demon/fiend be charmed by a dryad, if this could happen in the rules. If not is there a way to get around that problem? The fiend is a yochlol, which can't return to their original plane, and is having an angry fit about it in a forest and destroying the forest with what they touch.
According to the written rules, no.
A dryad's Fey Charm action specifically says it
targets one humanoid or beast that she can see within 30 feet of her.
A yochlol is a fiend (and a demon and a shapechanger), not a humanoid or a beast, so it can't be targeted by the dryad's Fey Charm.
Practically, however, you can do whatever you want to make a good story.
If you are the DM you can adjust the rules or just make the rules obey your story structure. The important thing is having a good time, not maintaining 100% fidelity to the rules as written. If that means that this yochol gets charmed by this dryad, just make it happen and don't worry about it.