I am new to 4E and am looking for a resource that provides all the rules in one place, instead of having to flip through a handful of books in order to find what I need. I see that Wizards has a print version of the Compendium as well as a Compendium available through the Dungeons & Dragons Insider subscription. Besides cost, what are the differences between these products? What should I think about when choosing which one to spend money on?
The Rules Compendium and the online Compendium are two different, similarly-named things.
The D&D Essentials Rules Compendium is a 2010 book that includes an updated version of the core rules that first appeared in the PHB and DMG. It includes the rules for combat, skill checks, awarding treasure and so forth. What it doesn't include is the game content: races, classes, creatures and so on. You're expected to use the online Compendium for that.
D&D Insider Compendium
The D&D Insider Compendium is an online searchable database of every item of game content from every sourcebook, Dragon and Dungeon magazine. Game content means things like races, classes, feats, magic items, powers, creatures, poisons and so forth. It includes the full errata-updated text, not just a list: we're talking thousands of items and creatures.
It doesn't include an index, and it doesn't explain the rules of play, although there is a glossary which defines any game terms you want to look up, like "skill challenge" or "combat advantage".
Subscribing to D&D Insider grants access to the Compendium as well as the character builder software, online Dragon/Dungeon magazines and other useful tools. You can search the Compendium for free, but have to pay for a subscription to see the content: $9.95/month or $71.40/year.
If you want all the feats, magic items and so on, the online Compendium is your best option. It will give you every feat, every item including latest errata which your book won't have.
If you also want a handy copy of the core game rules for reference at the table, the Rules Compendium book is a good choice. The PHB1 is outdated now due to errata. However, if you're on a budget, already know how to play, and you can reliably bring a computer to the gaming table, the online Compendium does include the definitions of each skill, combat option, and game term.
D&D Insider is your one stop shop
I've been playing 4e for a few years now and have never bought a single book. I've been able to do this because between the Character Builder, Rules Compendium, and Adventure Tools (monster builder, encounter builder, etc.) I've found everything I need as a player and a DM. You will miss out on some of the design philosophy imparted to you as a GM in things like the DMGs and the MMs, but if you already grasp the core mechanics of treasure parcels, encounter design, and the like then there is nothing else for the books to offer you that the D&D Insider can't give you in a far easier form.
All published 4e content is (or will soon be if it comes out between updates) be accessible on the compendium. That means you can easily give yourself and other players access to all the stats, character themes, backgrounds, feats, and items available.This is very important if you like to optimize and build towards specific mechanics or damage types such as charging, radiant damage, cold damage, etc. I also have to say that building characters through Character Builder is infinitely quicker, easier, and allows more comparison between options (feats, items, powers) then if you were trying to "flip through all the books" as you say.
Beyond all the books you'll also get access to all materials published in Dungeon (DM resources) and Dragon (player resources) magazines both to read as published and also integrated through updates to the character builder and compendium. It should be noted that while the compendium is a rules resource, the character builder is not as firm as it often lags behind the compendium in updates (things can take a few weeks to reach the builder) and there a few known glitches that were never properly resolved.