Since the question hasn't specified rules as written I want to include my own interpretation which I have used in my own games.
While the rule as written in the book says it takes a full action to don or doff a shield. Myself and my group has always interpreted 'doffing' as not only taking it off, but also stowing it. As a group we came to a consensus that throwing down a shield shouldn't take a full action unless the shield was expressly strapped to ones arm. This is where we looked into real life and shield design to get our inspiration, what we found is that there are 2 main types of ways to hold a shield:
One that includes strapping it to your arm as well as holding a handle.
A 'punch' shield, or one that only has a single handle.
The rules we have now say: While it will take a full action to drop a shield that is strapped to your arm, a simple punch shield can be easily dropped without taking a full action. This allows the character with a versatile weapon to drop a punch shield, change their grip on their versatile weapon, and attack 2 handed in the same action.
Picking the shield back up and re-wielding it still takes a full action.
In our experience this rule has still felt balanced and allowed for further combat flexibility. Allowing a character to naturally throw off a shield sacrificing some of their protection to hopefully deal a little more damage, and still leave them with needing to invest an action to reequip that cast off protection.
It has happened more than once that a character who throws down their shield takes a few strong hits that would have otherwise been blocked, and suddenly realizes the mistake in their action.