I believe it is a matter of story, and less a matter of mechanics. Mechanically, any monster, any NPC, any curse, any trap, anything the players encounter will have a solution, a stat to beat, and you as the GM would have calculated their chances and deemed it possible for them to defeat (speaking in generalities)
The way to make the Undead scary is not to increase their mechanical difficulty in the game, or change the methods that are used against them - making a skeleton suddenly not receive extra damage from crushing weapons only upsets the players.
In my opinion the only way to make something be scary is to have it cause fear. Since you yourself can't actually cause the players themselves to be afraid of a fictional monster, you need to make the undead really scare, and spread fear in the game.
Methods for Causing Fear
The first step to causing fear is hearing about it. The common folk are spreading rumors about strange disappearances a few towns over, an unusual amount of people leaving their homes without an explanation are seen wandering around the countryside, if questioned they might just "have a bad feeling" or maybe their livestock just turned up dead one morning, whatever it is, something is disturbing people, and no one knows what it is.
No one really knows who said it first, but once the idea was planted everyone was saying it - Undead. No one has any confirmation, but everyone is full of superstition and the commoners are no longer just grumbling about strange occurrences - they are gathering to demand answers, they are forming night-watch groups.
Clues and Panic:
With the rumors of strange things afoot, and the murmurings of Undead, people are more vigilant. Suddenly the strange footprints in the forest are no longer dismissed as unimportant- they are from the Undead. The boy who went missing? Undead, the noises at night? Undead, scraps of clothing on a tree, bloody marks on a wall - Undead.
The leaders are telling people to stay indoors after dark, mercenaries are hired, guardsmen are on double watch, adventurers start showing up in taverns looking for glory! All this, without any evidence, no proof of Undead yet - but the commoners, the leaders, everyone is afraid. Excursions are sent into the night to discover what might be lurking...
They never return. Bodies not found.
Panic and Fear:
Everyone knows now. There must be no doubt - commoners start packing up their things ready to flee, the rulers are trying to keep them, but even the guards and mercenaries are being stubborn. And yet - there has been no one alive that has yet lived to tell of the Undead.
That is when it happens. That is the night when they came......
And I suppose you can figure out what to do from there. Maybe the Undead kill everyone in town save a few survivors who manage to escape and tell of it (in all its horror) maybe that is when the rulers send for the heroes to come rescue them...
But whatever happens after all these stages, the fear is part of the world around them - any roleplayer worth his dice would know that his character should be afraid - even if he himself is not.
Once the Undead are actually part of the game, when the fights begin - then to cause them to really be frightening is to have them be unpredictable, forceful, and unyielding - just like a proper horde of undead.
Tactics for Undead
Have them be hiding, under ground, behind doors, in crates, on the cieling, drowned in the water, hanging from the windows - anywhere you can place a hidden corpse.
Numbers of undead beyond counting, beyond the skill of the party, too many to fit anywhere - but in the distance, coming slowly, the players have every chance to escape, but seeing those numbers, witnessing the hordes will give them pause.
Any fallen ally, any corpse, any innocent victim - have them become Undead as soon as you can, preferably while the players are watching, and as fast as you can.
Beyond these ideas, you could always add a few necromancers, liches, vampires, or other powerful leaders for the Undead that would give them tactics or a strategy (always something to be afraid of) Or just giving the Undead some big powerful monsters - But that is less in the idea of what I believe your question to be about - how to make a bunch of skeletons scare the players.