Generally, Old School refers to a style of play. Modern, when used for styles of play, has a much more nebulous collection of meanings. Modern, when referring to settings, generally means anything set post 1975 (a 10 year difference from the official historical definition of 1965 and on...).
That Old School style can be summarized with "Kill Them And Take Their Stuff" and/or "Prepared Environment." It also tends to emphasize "Let the dice fall where they may" and "Rulings, not Rules."
Most early games focused on the killing. While many groups didn't do so, the rules certainly did, and most of the early games look more like character scale minis games than a means of shared storytelling. The combat systems are usually almost purely battlefield action, whether it be mano a mano or army a army.
Prepared Environment, very much like the Montessori teaching method, you do lots of prep on the environment, and drop the subjects in to see what they learn by hitting the prepared encounters. The simplest to run is the classic dungeon; draw it, stock it, and coerce PC's into it. New school games often allow players to alter the setting on the fly, and frequently have methods of play written in that make most preplanning irrelevant.
Let the dice fall where they may... one of the attitudes often associated with "Old School Play," in part due to Gygax's admonitions to do so, means that once you roll the dice, you accept the results. If you attack the dragon, and roll a 1, you missed, now get ready for the GM to have him attack you. And if he rolls a 20, well, that's likely to be a critical hit... and a dragon doing a crit usually means time to let the dice fall where they may for the next character.
"Rulings, not Rules" is the motto of many in the "Old School Revival"... an attitude that it's better to have a light, simple set of vague rules and the GM makes rulings on the fly, rather than a heavy, comprehensive and detailed simulation ruleset.
The "Modern" school of thought focuses a lot on the nature of non-combat play. It's not a unified and codified group, but a clade of several related outgrowths of the 90's several styles. It's characterized by several different traits: Social Conflict mechanics, narrative structure as part of rules, new ways of looking at character definition, many delve into rolling or bidding for Narrative control, and many (including my own designs) have latched on to D. Vincent Baker's admonition of "Say yes or roll the dice," the "Yes, And" and "Yes, But" corollaries , and the Luke Crane axioms of "know your reward cycles," "write down what you want conflict over," and "Tell me what you want not what you try." Finally, many change who the final say is. (All of which see below.)
Social conflict is the most common "new school" element - extending resolution out of the realm of the physical and into social and emotional realms. Systems regulate those debates, so that the player who is a master debater with the CHR 3 character actually winds up losing, and the stuttering teen with the Chr 18 palladin actually wins them.
Narrative structure as part of several games is written into the rules. Rather than leave the narrative flow to the GM, many have quite detailed processes for narrative control. Some enforce scene budgets, others have shared GMing, and quite a few have ways for players to add things into the setting mid-session; Rather than making a knowledge roll and asking the GM, you instead make an assertion, the GM sets a difficulty, and if made, the assertion is true.
Narrative Control: A few games have a traditional GM, but instead of having him/her decide difficulties on actions of import, players either bid for, roll for, or some hybrid thereof, the ability to make the decision as to who succeeds and who fails. Some such systems can get rather involved... Several of John Wick's designs make use of this. You don't roll for success, but for the chance to decide.
Character Definition: many of the more recent games are doing different things in setting up how characters are defined. Some have no attributes, or attributes that don't seem to be innate. Smallville uses Relationships, for example. John Wick's Blood & Honor has no skills, and no way of raising attributes, but you have a raisable level in your job, and aspects that affect your ability to perform.
D. Vincent Baker's Admonition: "Say 'Yes' or Roll the dice"... often, many new school games point to "GM's shouldn't say 'no!'" If in doubt, go to the dice. And there are also ways to say Yes but not give in, such as "Yes, but ..." and "Yes, and then...". But, unless it's important to the story or implausible that they succeed, just let players have their characters succeed. Or let them succeed, and then something else mitigates it.
Luke Crane has several bits of advice peppering his very successful Burning Wheel and it's descendant games. Building on Vincent's advice, he suggests only going to the dice when failure is interesting. Luke also encourages people to know the reward cycles, and use them to drive the game forward, and to work them as both GM and/or player.
What you want, not how you try: Luke goes to great lengths, as do some other designers, about getting players to tell the GM what you want from a success. Rather than, "I attack the rope holding up the chandelier," instead, "I want to stop the duke from attacking the countess. I'll use my sword to knock down the chandelier, to trap him in it..." It's wordier, but it defines the goal... a failure then isn't leaving the chandelier up, but missing the duke, or hitting him but not preventing his attack on the countess.
What do you want to struggle with? This is the fundamental question in many new school games. In writing your character, you explicitly provide instructions for the "GM to poke here." Luke Crane makes extensive use of this in defining a character's "Beliefs and Instincts"... You define a belief and it's resulting action... and the GM uses that to bait you. And if you stick to the belief, you get a reward. If you instead act against it, you get a different reward. The FATE engine also uses a "poke me here" mechanic, called Aspects. Every aspect is a two edged sword - it can be used by you to enhance your character, and against your character to earn a reward....
Many also move final approval (of rules interpretations, of characters, of setting elements, even of bad guys in some games) out of the GM's hands, and to the group's hands as a whole.
No one game has all of these (tho' Luke Crane & Chris Moeller's Burning Empires comes pretty close).
In Between: in the late 1980's and early 1990's, a lot of very odd games came out. Most went nowhere, developing a few fans in a local area... but a few did things very differently, and made big impacts. Several are of note - they still go strong, in their fan-bases - and still get derided by many.
The White Wolf school of thought marries fairly traditional game mechanics to a story first mentality, and adds ways to bypass rolling the dice. Rich settings, rife with conflict, and setting books written from a non-omniscient, in world point of view, often contradictory to other sourcebooks.
The Rolemaster Mode: Tables for everything. Rolemaster gets started in 1981, but really hits its stride in the very late 1980s - 1988-1992. Tables for every weapon, lots of details to look up. If you can find the right tables when you need them, it's a fast and fun game... but it tends to bog down with new players due to the massive piles of tables. (Last time I ran it, for a party of 4, I, as the GM, needed about 20 tables to hand, each 8.5x11, and the players each had their one 1-2 weapon's tables copied and in front of them.)
Hero, GURPS, CORPS, Masterbook, EABA: The Universal Systems. Several games arose in the late 80's and early 90's using a single unified game engine for multiple settings, and use point based characters. EABA, one of the most recent, I happen to have playtested. Hero was one of the first point based systems, and Champions 1st ed was noted for incredible flexibility... so much so that they adapted it to other settings (Justice Inc, Danger International, Robot Warriors, Fantasy Hero) each incorporating a slightly modified version of the rules, and later released it as a core rules set and separately sold world books. GURPS started with Core and Worldbooks, each sold separately, and still works that way. Masterbook was almost always "Bundle the Core Rules in the box with the worldbook".
Palladium, BRP, Cortex, LUG's Icon, Decipher's Coda, GDW, and FFG's games- The House Engines: Several companies use the same core rules in all their games, but adapted to various settings, tho' not much; they reprint the core rules in each game. Palladium, once you learn one game, the only things differing are the magic lists, classes and races available, and sometimes a few genre emulation rules. BRP was skill based, but otherwise, each game included the same basic engine, but additions made to tailor it to the setting of that particular game - until recently, that is, when it went to core + worldbooks. GDW used the same engine with slight tweaks for Twilight 2000 second edition, Dark Conspiracy, Cadillacs and Dinosaurs and Traveller the New Era. Last Unicorn released 3 Star Trek games and a Dune game all using their Icon system; each was stand alone, but each used exactly the same skill and combat mechanics. Decipher only released two: Star Trek and Middle Earth, and there are slight differences between the two, but if you know one, the other's a quick pickup. The currently in print 40K RPG's by Fantasy Flight (Dark Heresy, Rogue Trader, Deathwatch, Black Crusade) are another example of house engine, but one where character generation and growth varies, especially the power level and psionics rules, but the majority of the mechanics don't change. By comparison, FFG's Star Wars line is identical rules, save for a key flavor rule, and different classes, opponents, vehicles, and force powers in the (currently 2) core rulebooks. Note that FFG's Star Wars line uses a very close variation on their WFRP 3rd Ed system.
