Dronz's answer does a very good job of covering specific ways to keep the unknown feeling like it's unknown and not damage setting cohesion with questions like "but why doesn't the US Government know about the Mi-Go?"; but to state some more general principles:
- People, by-and-large, don't believe in the supernatural and certainly don't know anything about the Mythos.
- Supernatural events are often very difficult to believe as having been supernatural events if you're not personally experiencing them (sometimes, even if you are).
- Supernatural events generally happen to individuals, not to governments or organisations. This makes it very difficult for anyone other than affected individuals to even know about the existence of the supernatural.
- Individuals who do find things out about the Mythos tend to suffer severe consequences, even the ones who don't die or go insane.
From these axioms we can derive a theorem that can help explain how a modern-setting game can both be isolating (for the players) and maintain unknown-ness (with regards to the threats):
Facing supernatural threats is an inherently isolating experience
Let's be clear here: in stories like this, horrible things are going to happen to characters, and it's going to fundamentally change them. Even if they escape physically unscathed, the mental anguish of even a low-grade Mythos encounter - like, say, a flesh-eating ghoul - is not to be taken lightly. The newspaper reporter investigating weird noises from the graveyard is a fundamentally different person from the man who makes it back to his car shaking, with scratches all down his arm and gunpowder residue on his hands, thinking about how the thing raised itself back up even after being shot.
And unfortunately for him, he has changed in a way that makes him less able to relate to his blissfully unknowing colleagues. It's not like he can tell them the truth - not the whole truth, at least. He won't be believed. Which is not to say that he will be dismissed out of hand, but:
- outlandish claims require outlandish justification
- making outlandish claims and not being believed is psychologically taxing
- there are probably non-supernatural explanations for the physical evidence (if there even is any)
Some might be inclined to believe his story, perhaps even to believe that he didn't make any mistakes or misremember or exaggerate details (something that eyewitnesses do all the time in real life). But even then: they didn't actually experience it. On a very real level, he can't relate to them about it.
And some, especially if they're not close to him personally, will disbelieve. They will, at worst, think him a fool or a crackpot. Every time he tries to draw someone in to the world of the Mythos and they resist, it damages his relationship with them.
After not too long, the only people that a Mythos investigator can truly confide in are other Mythos investigators.
In real life, soldiers and similar professions often experience this form of alienation without anything supernatural being involved. But wait, it gets worse:
In order to maintain any amount of social contact with normal people, secrets must be kept.
Faced with losing the trust of people he tries to tell the truth to, investigators could just stop trying to illuminate their Muggle acquaintances. After all, it's highly useful to not be a social pariah. But this ultimately leads to keeping secrets from people ("oh, those wounds are from... a car accident."). Keeping secrets is isolating in and of itself (it's hard to connect with someone when you can't let them in) and also tends to be somewhat unsustainable - unless you're a master manipulator, people will eventually figure out you're not telling them the whole truth.
And if you keep on investigating the Mythos rather than, say, seeing a psychiatrist, and consequently you keep having to explain to your family how your shirts keep getting blood on them and your sudden interest in high explosives, then you're going to quickly burn through their trust until you have no friends.
Speaking of which:
The Mythos will draw your friends in if you let it
The other reason not to tell people about the horrible things that you face on a nightly basis is that if they do believe you, that means they're not a bystander any longer. That means exposing them to blood, death, madness, incomprehensible horror and the true insignificance of humanity in a cold and uncaring cosmos. If they are a person you care about even slightly, you've just put them in great danger. They might be able to back out now - forget everything you told them, disbelieve, and cut off all contact with you. Otherwise, they're on a path that leads to a padded cell, suicide, or a death beyond description.
Not something any rational person wants to inflict on friends or family. And yet it could happen regardless: you don't need to tell someone the truth for them to figure it out. Simply associating with Muggles puts them at risk of being drawn into the Mythos. Investigators who value their friends' humanity have a strong motivation to cut ties.
But enough theory!
The question asks for:
tried and tested approaches to solving these problems when writing scenarios and campaigns for modern investigative/horror settings.
The chief approach I have to recommend, in keeping with the arguments laid out here, is simply to cultivate the principle that having experience with the Mythos is destructive and, more importantly, sharing it is ineffective. This is in fact the exact same principle that applies to non-modern investigative horror, it's just that it comes a bit more sharply into focus in a modern setting.
Consider Call of Cthulhu's venerable introductory scenario, The Haunting. As it's written, there seems to be no reason whatsoever why it shouldn't play just as well in a modern setting as it would in the 1920s.
There's a supposedly haunted house; the players have been hired to investigate it. There are several vague yet foreboding accounts of supernatural events, yet despite at least one death and multiple insanities, there's certainly no worldwide interest. Only a handful of individuals have personally experienced anything definitively supernatural, and most others (let alone organisations such as the police) certainly don't believe in ghosts enough to consider investigating them. Unknown-ness is preserved without any particular need for a narrative device. People just don't believe in the supernatural on such shaky evidence.
If you want to apply this technique to modern games, just accept that nobody except cultists and unlucky investigators really knows anything about the Mythos. If you don't like the conspiracy-like vibe of Delta Green, the only solution is that there isn't a grand conspiracy. There is no secret police force that has been monitoring Nyarlethotep since 1935. The government doesn't know about the Mi-go because the only scientist who even figured out they existed got fired when his official report was just unintelligible symbols scrawled in his own blood. The first and only line of defense against a threat nobody else can even imagine is the 4-6 poor schmucks who were in the wrong place at the wrong time because they're the only ones who know anything at all.
What about isolation? Again, in The Haunting, no device seems necessary. The PCs can go wherever they wish. They feel isolated when in the house because there's nobody around, but we don't expect that the entire street is deserted. They can tell people about the hauntings they see, but it's not like claiming to see something spooky in a supposedly haunted house is unheard of. A PC could use their mobile phone to call for help, but they'd likely have more on his mind when faced with a magical levitating knife stabbing at them. And how would anyone they called be of much help?
In the scenario, if the PCs leave without resolving the mystery, the next night the house's owner stays overnight and is killed in his sleep. If they want to save him (or possibly avenge him), they have to face the thing in the haunted house themselves, alone in the dark basement, because doing otherwise means letting innocent people who aren't prepared to face it do so.
If you want them to be isolated, you can still let them talk to people. Just show them the consequences of getting people involved.
Okay, but what if there's this one situation where the contrivances of the modern setting really do ruin the tension? Or where losing the contrivances of the modern setting could enhance the tension in an interesting way?
Well, honestly, the answer to this is the fairly boring "then you need to make the modern setting's contrivances not apply." In most of this answer I've fairly strongly championed the "communication is ineffective" strategy, but if you want to use the "communication is prevented" strategy, there are a bunch of ways to do it.
- Situational contrivances (e.g, the PCs must visit an oil rig in the middle of the Pacific Ocean.)
- Physical contrivances (e.g, mobile phones short out and die in the presence of magic)
- Drawbacks (e.g, looking up the Esoteric Order of Dagon on Wikipedia gets you misleading information planted by the cult)
- Supernatural negation (e.g, eldritch beings cannot be recorded on video, it just shows an empty room)
- Supernatural hostility (e.g, your computer screen displays a giant eyeball that nobody else can see)
- and probably many others beyond the scope of this answer (otherwise it would become an answer to "How to create isolation and the unknown in horror games?")
Done well, all of these techniques (the preventative ones I just listed and the chief one of making communication ineffective) can result in a modern horror scenario being even more effectively isolating than an antiquated one, because in a modern scenario the players expect to have certain abilities and powers, which means that expectation can be subverted.
I once played a horror computer game that trapped you in a hotel simply by having it always be the next building you found on the road, no matter how many times you walked past it. It was amazingly unsettling, because there was no invisible wall. They gave you the power to leave. It just didn't work.