Delta Green* (The Standalone Game)
Delta Green is a modern-day scenario where the players play agents of either an ultra-secret Department of Defense special access group or a rogue criminal conspiracy, who fight a secret war against utter annihilation at the hands of Lovecraftian horrors, pushing back the seconds of the doomsday clock at the cost of whatever they hold dear.
Supports Modern Day
Delta Green is current up to past 2010 in its lore. Everything you will find is essentially modern-day, whether it be weapons, universe lore, and hooks.
Works with a Custom Setting
Delta Green is very open-ended. It has some pregen campaign hooks, like the one in the Agent's Guide, but you are by no means required to follow those. In fact, the authors go out of their way to encourage GMs to mess with various aspects of the game, including the setting. It's a horror game, and the last thing you want is the surprise to be spoiled at the beginning.
Magic should be difficult/time consuming
There is a list of suggested rituals in the Handler's Guide (with the usual caveat that you should feel free to modify them as you see fit), along with descriptions as to what they do, what ingredients they require...
Oh, and the cost to your Willpower and Sanity for performing them. Because what humans call Magic is science to the Old Ones, and each glimpse of understanding into things Man was Not Meant to Know brings you one (or two, or several) steps (or flying leaps) closer to the abyss.
Classless
Mostly. The game uses skills and careers, which are both very important, but there aren't class-specific abilities or anything like that. The key places where careers come into play is both in realistically determining what skills a character might be able to have (for example, a run-of-the-mill CDC official would not have skills in, say, Hacking, using Mortars, or thieving), and what reactions the world might have to a character doing certain things. For example, an FBI Agent can arrest someone, but a CIA Agent or Private Investigator often can't. Likewise, if someone saw a CDC or EPA agent packing heat, they'd get a little concerned and start asking questions.
Investigations/Mystery take priority over combat
As Delta Green is a horror game, the authors actively encourage GMs and to focus on investigations and story, with combat as a tool and not the entire game. GMs, likewise, can do this to their players in a number of ways. For one, a dead suspect can't give information, and also the game's realistic setting is going to punish excessive belligerence because it will call attention and heat down on the Agents' heads, potentially getting them arrested, suspended, or crippled by being under extensive review.
Furthermore, some entities just can't be hurt physically by mere mortal weapons. A quick-thinking desecration of a summoning site, the solution to a particular equation, or the completion of a certain ritual might be the only thing that can keep the mortal world separate from rampaging horrors.
Private Investigators, not Military
The game is set-up with Law Enforcement in mind, but the illegal situation of the Delta Green Conspiracy lends itself very much to having Private Investigators rather than Military Operatives. In fact, the Agent's Guide even has a recommended quick-start build for PIs, as well as a substantial range of other civvies such as Socialites, Criminals, Professors, and Programmers.
Ideally, the Supernatural Parts can be Hidden until they crop up
This is very much the nature of the game and the lore. The Supernatural isn't a club to beat the story with, or even the cake-batter. It's used sparingly as a twist, and it's a pervasive fear, both in the game and in the meta, but overuse of the Supernatural desensitizes the characters and the players to it, which ruins the atmosphere. If you consider Alien: Isolation and how it treats the Alien, you'll see what I'm getting at here. If all you did was run away from the Alien, it would start to feel pretty stale.
Playable in Roll20
I don't think Roll20 has the actual system yet, however, it is a pretty standard d100 system. If you've ever played the FFRPGs, just toss in the Lethality rules and bam you're good. I'm not sure that Roll20 really captures the atmosphere of Delta Green as well as in-person does, but it can make certain aspects of the game easier.
Online Character sheets
I am not sure if anyone has online sheets for Delta Green, but the company has released PDF copies for free.
*Another answer suggests the Call of Cthulhu Delta Green supplement, which still uses the C0C system, and mentions the Standalone as a beta, which is no longer accurate. The Standalone is its own system, current up to past 2010 in its lore. I have linked to the Arc Dream store page where you can buy it, but you also can get it from Amazon or in a game shop.