If you've seen the movie Ip Man, you'll remember that he used long wooden implements in a way akin to a staff, yet focused on quick, precise strikes, rather than brute-strength bashing. What's a balanced way to house-rule a staff that uses Dex (i.e. has finesse), for non-Monks? Presumably a fighter, given all his/her training, should be able to pull it off (imho).
I don't think so.
The problem I see here, ultimately, is that this would give access to a two-handed finesse weapon with a d8 damage die. No finesse weapons are versatile, and the only d8 damage die is the rapier.
That might not seem like much, until your Rogue multiclasses Fighter and picks up GWF and now is able to reroll 1's and 2's on their sneak attack dice in addition to their weapon dice. The concern here is that this provides a rogue with a weapon that will add, effectively .6 to every sneak attack die they roll (avg of 4.1 vs 3.5).
Let's look at this in light of other options available:
Simple finesse weapons:
- Dagger d4 (light, thrown)
Martial finesse weapons:
- Short Sword d6 (light)
- Scimitar d6 (light)
- Rapier d8
- Whip d4 (reach)
The weapon you're proposing would be finesse as well as
- Quarterstaff d6 (versatile (d8))
This means that the staff would fall into a group has a single finesse option, with a much smaller damage die, and no two handed option. You might be able to get away with it as a martial weapon, but I still don't think it's a great idea. The extra property that allow for the d8 would make it strictly better than the rapier. It would basically become the weapon of choice for Rogues if they could get it, and would become a very strong option for a light armor fighter (though not as strong as the rogue).
Out of turn attacks:
Quarterstaff qualifies for the Polearm Master feat. This means that you'd be giving a Dex based character access to a pretty high powered feat. Again this is going to point at the rogue who would be taking a huge amount of advantage of this and probably inflicting sneak attack a second time most turns. Basically this would also let other classes who don't get armor proficiencies and want to engage in polearm shenanigans boost Dex instead of Str.
In general, there aren't any huge reasons why this wouldn't work. Some potential pitfalls:
You give a powerful melee weapon to casters - By making a simple melee weapon finesse based, you are encouraging casters to dump Strength and push more points into Dexterity than they already would. Functionally, this would eliminate the need for melee cantrips, and possibly draw away from other members of the party.
You draw away from the monk - If your party has a monk, then you are taking away a cool feature of their class. Granted, it is a small one, but by giving finesse attributes to quarterstaffs you are removing some of their specialness.
Quarterstaves become extremely versatile for casters - Most people I know have houseruled that quarterstaves can count as an arcane focus, though in the rules it appears they are separate. If you double up, then quarterstaves become a consistent source of damage. Furthermore, if they get the Warcaster feat, you have an incredibly powerful, well rounded character as a wizard.
Possible abuses exist for rogues - As wax eagle mentioned, Rogues could multiclass into fighter, gain Great Weapon Fighting, and have an extremely powerful sneak attack.
Overall, I would discourage it, but there are definitely situations where it might make sense. If a character wanted it for flavor (it's just their walking stick, nothing to be afraid of) and you trusted they wouldn't try to game the system, go for it. However, anytime you introduce a house rule be prepared for it to bite you later on.
I may be misunderstanding your question, so I'll first state it how I understood it. Your primary goal is to allow non-monks to use a quarterstaff with precision attacks (typically demonstrated in the game through Dex) without becoming unbalanced. If that's the case.. Fighter already has a mechanism for that, under the archetype Battle Master the Fighter is able to perform the "Precision Attack" maneuver. Granted it does not work exactly like the finesse attribute by allowing you to use dexterity rather than strength. However, the fighter does, for all intents and purposes, have the capability to perform calculated precision blows through this mechanism, and still maintain the ability to use (what generally is) the fighter's primary attribute.
That said, there exists the Martial Adept feat that then allows any class to gain the ability.
A second option is to mark it as a finesse weapon.. I would make it so its only when used two handed because really, think of someone trying to 'finesse' a quarterstaff one handed.. Thats silly.. Doing so would have the item treated like this: It's either versatile and you use strength modifier for it (unless you're monk). OR it has finesse but it has to be two-handed (thus negating the GWF problem).
Though I think I'm the only one in the world that likes conditional rules.
Personally, I think that only the monk should have the ability to finesse a weapon like that as doing so is all about moment and centering oneself. If a real fighter were to learn those abilities it would be by training as a 'monk'. That's just my take on the whole thing though.
You could subtract 2 damage, making it 1d4, or 1d6 when two-handed.
This would make it closer to the damage profile of a dagger, but instead of having the light and thrown properties, it retains being versatile. Or like a greatclub, but with 2 damage traded off for finesse.
The reduced damage might slightly offset the overpowered advantage that the Rogue / Fighter combo gets, described in Wax Eagle's answer, but that situation still seems overpowered. Maybe rule that your weapon isn't eligible for GWF, but at that point you're making things complciated :)
I have been thinking of this... right now I have a hexblade /pact of the blade that uses a glaive (sythe for flavor) with the polearm and sentinel feats, and will be multiclassing to swashbuckler. The solution that my DM and I came up with is to use a modiffied version of the Improved Weapon Incantation to give the pacted glaive the finesse property(witout the other +1 stuff and only within a 5 ft range).