You will be dead weight
Look at it this way: on level 161 you will have the AC of a first level Wizard, and they are not famous for their great defenses.
The main issue is lack of an appropriate AC, which makes the character completely unsuitable to be in the front liney. This were not so bad if you could always stay in the back row, but most DMs I know will use smart and fast skirmishers and archers to hurt the back row from time to time.
Xanathar’s Guide to Everything helps a lot here, in the PHB all damaging cantrips had a range of 30 feet or less. With the new options you can stay out of most monsters' walking range.
Not using weapons is a much smaller problem, it should only come up in antimagic fields, and those are very rare.
Other gear can easily be substituted by Wild Shapes and spells.
Climbing kit: spider form
Lantern: Produce Flame
After Monks, Land Druids lose the least by denouncing armor and shields, unfortunately this is not because they are fine without it, but because they [start from a bad situation] to begin with.
A first level Paladin is expected to start with AC18, Chain Mail+Shield. The Druid's closest cousin, the Cleric can start with the same, if the chosen domain provides heavy armor proficiency, or Scale+Shield+Dex 14, which adds up to the same final AC.
A usual Druid starts two lower for AC16, as the best possible armor without some homebrewing is Studded Leather. You start at least three below that. With point buy the highest Dex you can get is 16, and you do not use shields. So your final AC is 13, completely unfit for the frontline.
The typical CR1 monster has a proficiency bonus of +2, hitting the average Cleric with 25% chance, you with 50. Meaning you go down twice that often. More accurate monsters finish you even faster.
If you are willing to give up on Primal Savagery, you can stay behind in relative safety.
You can trade your defensive problem for an offensive one with this spell. Spending a 2nd level slot, you can get the AC of a first level Cleric, who does not use a shield. It is far from strong, bordering on useless even, based on only this.
What makes it worse is that it requires concentration, preventing you from being effective in combat. Your arguably best offensive spells require concentration: Faerie Fire, Entangle, Flaming Sphere, Call Lightning, Guardian of Nature and so on. So for a mediocre AC you give up most of your combat capabilities.
You said you want to increase your Dex to compensate for the lack of AC, unfortunately you could only do it again at the expense of your offensive abilities.
Assuming a reasonable ability score distribution of Wis 16, Dex 16, if you do not want to prioritize your defensive ability (Dex) over your offensive one (Wis), you will reach Dex 20 by level 16. Again, your AC at this point is one point lower than what a 1st level Druid is expected to have (Studded+Shield+Dex 14).
With the Magic Initiate feat you can use Mage Armor to increase your AC by 3. However, unless you play a variant human, this will postpone your Dex 20 to level 19.
This build is unusable in the front line, Primal Savagery has to go.
You can only stay far enough from melee enemies if you use the spells from Xanathar’s Guide of Everything, Druid cantrips from PHB have horrible range. With XGtE this build becomes acceptable, but just barely.
Your character development (in the literary sense) could involve slowly and reluctantly realizing that it is tool use that makes humans competitive with animals. We lack the though hide of crocodiles, the fangs of tigers and the strength of bears.
It would be an interesting aspect of roleplay to grudgingly start using these things.
I know want to avoid multiclassing, but it could be very useful and thematic.
You can take a level of "tought-by-wolves-how-to-use-your-senses-to-avoid-attacks", and approximate it with a level of Monk.
1) if you start with 16 in Dex and Wis, and increase Wis first, then Dex