If a character had only one hand available, but they were holding a two-handed weapon, what would the penalty for the attack and damage be?
I'd imagine the result would be significantly less effective than using a proper one-handed weapon. The only ruling I can find is that you need two hands to wield a two-handed weapon, but it seems a bit extreme that you can't even try to make a one handed swing.
Everyone has mentioned rolling with disadvantage and it makes a lot of sense especially when you consider the video from Icyfire’s answer demonstrating how a trained swordsman struggles when trying to use a two-handed weapon with one hand.
There are several weapons that already have the versatile property, which means they’re supposed to be able to be wielded one-handed. These weapons still go down a dice type when you do this, though, to reflect the reduced power you have behind them.
The thing with adding disadvantage alone is that, for instance, a gnome could wield a greataxe in the dark with just one hand, with another hand free for anything else, and despite the small size and lack of proficiency and everything else the only penalty they’d face is disadvantage. Have them successfully hide and it’s a straight roll!
Perhaps you could have a minimum strength requirement: only a character with a strength score above 15 can actually wield a two handed weapon with one hand; or maybe you homebrew a feat that allows this. In that case you might not even need to add all the other penalties...
Losing the proficiency bonus entirely is a pretty intense penalty, especially at higher levels. It seems reasonable that if someone is proficient with a greatsword they’d be passable, at least, when using it one-handed. However, if you try with the other two elements and it still feels unbalanced, you could halve the proficiency bonus, for instance (rounding down as you do with everything in 5e.)
As you point out, the PHB simply states that you cannot use a two-handed weapon with one hand. However, there are some real-world examples of people wielding a two-handed weapon with one hand. For example, this youtube video shows a trained swordsman using a huge zweihander with one hand, though he clearly has some difficulty. From this video, you can see that the sword can still be used as a sword, albeit more cumbersome. Therefore, I would rule that it's still more effective than an improvised weapon.
Also, ruling that it's an improvised weapon is a little weird--you're saying that picking up a greatsword is mechanically equivalent to picking up a tree branch. I know that 5e doesn't have much granularity in its rules, but it still doesn't sit well with me.
Consider the costs and benefits that a character gains while using a two-handed weapon. The two-handed weapon does, on average, only 2 dmg more than the one-handed variety, and they give up a shield or spellcasting. Assuming that they cannot benefit from the relevant fighting style or feats, applying disadvantage seems appropriate. Disadvantage is still a significant penalty, so this ruling won't incentivize long-term use of two-handed weapons one-handed, without being as punishing as forcing it to be an improvised weapon.
I'd rule the following: in one hand, the two-handed weapon becomes an improvised weapon, losing the proficiency bonus on its attack roll unless you have the Tavern Brawler feat. Since it does not resemble a [one handed] weapon, it deals 1d4 damage on a hit. It keeps its reach property as normal if applicable.
I would do the same as if a small character used a heavy weapon, or a medium character used a large weapon (as from an ogre or such). He attacks with disadvantage.
I don't think more than that is needed for game balance. Disadvantage is a pretty strong penalty, equivalent to about -5 to hit, while using a 2-handed weapon in one hand is really not a very big bonus, equivalent to about +2 damage and perhaps reach.
Let them roll with disadvantage, remove their proficiency bonus (as it's not really how you learnt to use the weapon, so how can you be proficient?) and possibly reduce their ability modifier on hit if you feel it should only be a last resort technique.
The most important thing with this one is to just go with your gut and make sure you stick by it throughout the game!
I feel like they would definitely lose any proficiency bonus, but also they should have to make checks for maintaining a defensive stance with an unwieldy weapon. So instead of disadvantage on attack, they make an athletics check or opponents have advantage until next turn. I feel like it’s more disadvantages from how you defend yourself that round waving a big weapon in one hand, and strong characters should be better at that type of thing anyway. Center of gravity could be thrown hard at an opponent accurately but set you off footing that round.
I would definately roll with disadvantage because it is a heavy weapon making the attack (in real life) slower, but i would not take the proficiency bonus away because (again in real life) when training with a weapon many practitioners often use a weapon intended for two hands with one. the concept is that it is harder to use with the one hand so when training it reinforces good form if you handicap yourself thusly
My own personal House rule would be that the First couple rolls (what ever their Con modifier is are free and clear) with just disadvantage. After that I would start requiring a con save vs exhaustion. With the DC steadily getting harder every round by 1.
I have not used this exact rule in this exact setting of events. I have used very similar rules in similar settings. The added downside of Exhaustion makes it so that while players may still attempt this tactic they do so in a very limited fashion
