You probably need to first see, and then harm, the night hag. Sadly, plane shift will not work very well -- you'd shift several miles when you crossed over. So whether you can just directly adventure on the ethereal plane is kinda up to your DM.
Sight
Detect invisibility will certainly allow you to spot the hag, but it has a range of personal and is not a divine spell.
True Seeing is available to clerics, but is obviously higher level and more expensive.
Invisibility Purge would probably work? Unlike the other two, it doesn't specifically say it reveals ethereal creatures, but it does clearly say "Anything invisible becomes visible while in the area." So double check with your DM on that one.
The general strategy would then be to have your wizard keeping watch with detect invisibility, since it has a longer duration. A cleric could then cast purge once the hag has been spotted.
Harm
The high SR and hard to dispel etherealness might make this pretty hard to pull off using only the core rules.
Blink is the best (only?) cheap option for fighters and rogues. You have a miss chance, but that's much better than having no chance of hitting. (Ghost touch doesn't work against fully ethereal creatures, only incorporeal creatures on our plane.)
Magic missile and spiritual weapon are two CR appropriate force effects in the core rules. Unfortunately, both allow SR. Fortunately, you could prepare an awful lot of them.
Sepia snake sigil is a force effect that doesn't allow SR, and would immobilize the hag if it she failed her reflex save. (Hags have a +9 Ref save.) You'd have to somehow trick the hag into reading something, though!
Outside of core, Force Orb is a no-brainer. It's a 4th level [Force] effect that ignores spell resistance.
There's also a metamagic feat, Transdimensional Spell, that would let you affect the hag with any spell. Getting scrolls with transdimensional versions of no-SR spells would be another tactic.
Some magic items would let you become ethereal, but that's a pricey option:
- Scrolls of ethereal jaunt and etherealness cost 2275 and 3825 gp.
- An obsidian steed costs 28500 but is reusable once per week.
- A cloak of etherealness costs 55000gp.
Everyone into the Hut!
Leomunds Tiny Hut is ambiguous on whether it keeps creatures out. If so, it's ideal for sleeping in, safe from the night hag's dream attack. (She'd need line of sight.) She could still attack you normally, but that would require becoming corporeal. And since she couldn't see you, she would be at a huge disadvantage.
But, if it does keep creatures out, it's a pretty powerful spell. So I might not allow it as a DM.