A lot of this relies on your presence as a DM, and the trust and willingness to be scared among your players. At the end of the day, if they want to joke, they're not going to be afraid, no matter how spooky your scenario is. A lot of times people joke because they're afraid or uncomfortable.
But there are things you can do that will assist people who want to be scared in being scared:
- Turn down the lights (but not so much you can't read your dice/character sheet)
- Start the session with some creepy music (Bach's Fugue in D Minor is always appropriate for Ravenloft)
- Use a sound app with appropriate noises, e.g. wolf howls, sinister laughs. Use only at appropriate moments. Don't overdo it.
- Candles. Lots of candles.
- Ask your players if they want to be afraid (maybe not in so many words). Get them to be excited about being spooked.
- Be descriptive. Tell the players everything about the environment that (from their perspective) would be able to be taken in by their five senses. "You smell fetid water, and hear a solitary drip coming from somewhere in the darkness of the cellar."
- Avoid naming monsters. If you tell the players they see a "rotted corpse shambling toward them, its facial features distorted by years of being gnawed by rats" they're gonna know it's a zombie, but as soon as you say the Z-word, they're going to stop picturing it in their minds and revert to what they already know about zombies.
D&D Fear, Madness, and Horror
I'd use these rules in any horror campaign with the 5e rules. Essentially, your player may not be afraid, until it's clear their character is on the line. Thus, giving them mental hit points may be the way to go. Make it clear to them that, although their character may not necessarily die, at the point that they lose all Sanity, the character does not belong to them anymore. It becomes an NPC at that point. That should ratchet up the tension.
A softer approach is to use the Madness conditions on pg. 258. Madness always confers a degree of control to the DM over a player's character, and thus is a situation which involves a lot of trust in the social contract. Otherwise, a player may either become very upset, or (worse) cease caring entirely about the character.
Y'know, back in the day, the scariest thing in the Monster Manual was the Vampire, because a Vamp could not only kill you, it could drain levels.
Here's the problem with monsters: You put one in front of a party of PCs, and they're going to want to kill it. They're going to assume they're supposed to kill it. You, as the DM, to the players, should warn them that not every monster is supposed to be killed, and not every fight is winnable. Moreover, you should use powers for your Monsters that enforce this.
This can be as simple as a Fear or Madness saving throw (Wis or Cha) every time you encounter the horror, whether it's the weeping angel suddenly being right behind you, or when they read a tome that explains the horrors of the outer dark from whence the monsters have their origin. If you're using Sanity as an ability score, that's mental damage. Otherwise, just a Frightened or Mad Condition.
You also need to determine if the monster is killable, and when. The usual trope in horror fiction, is that the monster is unbeatable, until you understand how to beat it. For the most basic version of this, give it invulnerability until a certain (custom, designed by you) ritual is performed, for example. Otherwise, the monster might simply be vulnerable, but very tough to beat. You need a stake to kill a vampire, silver to kill a werewolf, etc... Your monster should also have a weakness, if it is designed to be fought. If it isn't designed to be fought, you are playing Call of Cthulhu now, and you can put away your DnD manuals.
Isolation and Helplessness
Horror as a genre relies on isolation and helplessness to convey fear. If you're with your buddies and you can talk about problems, and joke around, you won't be as afraid as if you are alone. Likewise, if you have a magic, vampire killing sword, you're much less likely to be afraid of vampires.
DnD is not a horror game. It is a game about teamwork and empowerment. Thus, it is difficult to truly make people afraid in DnD. You should try dirty tricks like separating the party (e.g. getting them while they're asleep in different rooms, or falling through a trapdoor, etc...). You should also try taking away their ability to fight back (e.g. the cleric's god has no power here, a harpy makes off with the paladin's sword o' smitin' evil).
But you're going to have to balance this with the fact that this is not what DnD is about, fundamentally. It's about going on an adventure. People are going to come to the table with certain expectations based on the tropes that DnD represents. They're going to expect to win. And, since it's DnD, you need to make it possible for them to do that. But you also need to make it seem like the odds are stacked against them.