What kind of game is a roleplaying game?
No matter how many times this is asked, it's always a tough one to answer. A roleplaying game is a fascinating mix between a bunch of other games and mediums you're already familiar with.
At its core, roleplaying is probably most strongly linked to children's games of make-believe. Think of playing Cops and Robbers, or playing out new stories with your favorite heroes, as portrayed a collection of action figures. In games like those, asking "how do you play this" feels almost incongruous - you just kind of know what is you're making believe, and the fun is in playing it out, in whatever way is handy, exciting, and fun.
The other huge influence is group storytelling. Imagine a game that goes like this: you sit down with three or four friends. One of you starts telling a story, and goes on for a couple of minutes. Then the next person picks up where the first left off. And around the circle you go, each one of you adding your own bits and twisting the story in the direction that interests you most.
There are other influences, but let's stick with those two for the moment. The traditional roleplaying game goes something like this: a bunch of friends get together, and they all want to play a make-believe game together. They're going to pretend they're heroes or wizards or pirates or policemen or anything else. And the way that the game works, when you boil it down to the basics, is very simple - each player has his own character, the person they're playing. And each player always lets everybody know what his character is doing - that's the game, they sit around the table, and create some story scenario, and then they just play through it, start to finish, with each player filling in the details for his own character.
One player won't have his own character; they're the game master, and they're job is to give the heroes a story to play in. Most crucially, he's typically in charge of the bad guys in the game, the people the players need to deal with. I won't get into the game master job at all; I just want you to understand that there's one person whose job it is to keep the story going, and to fill in all the detail besides the character stuff all the players are doing.
What's Dungeons and Dragons (and what are the books for)?
I've explained the basic format of a "roleplaying game," now I'm going to tell you that there are dozens and hundreds of roleplaying game systems. A "system" is a set of rules by which a game is played. Dungeons and Dragons (abbreviated D&D or DnD) is the best known roleplaying system, so I'll use it as an example of what a system is for, and what kind of rules you need for a make-believe game.
- A system usually establishes setting and/or genre. D&D provides a game of high fantasy, heroism and magic. The system explains how to play dwarves and half-elves and sorcerers and holy clerics. It's not built to deal with gunfights or academic rivalry or a million other subjects; it doesn't tell you how to play those within the system.
- A system establishes rules for conflict resolution. Stories are always all about conflict; tension comes from not knowing whether our heroes will succeed or fail at the next thing they're trying to do. So the rules decide whether or not you can cast a particular spell, or how hard it is to kill the enemy archer, or which of two characters win in a fistfight, and which one of them manages to woo their mutual beloved afterwards. Whenever you get to a point where you say, "I want to do X," and you know you won't necessarily succeed - the system provides rules to figure out what happens.
- A system provides initial material to work with. You can't play in a vacuum; everybody needs to have some sense of what the game setting is like, and what options they have as players. So D&D goes into great length about different types of characters you can play, different abilities you can have, what monsters and spells and magic items are in the world, and where they fit in with the game rules. For example, you've got entire books that are compendiums of different monsters; the game master uses this to have enemies to throw at players without needing to make it all up themselves (and not needing to hopelessly guess how many pirate zombies it would take to give your particular group a fair fight).
So that's where the books and the other apparatus come in - every game needs rules. Some games use dice (which puts randomness into the "can my character do X" question), some use boards (to keep track of a complex battle scene), D&D frequently uses both. And all but the simplest games have some kind of rulebook - and often, additional books chock-full of optional additions. So if you're playing a game based on Buffy the Vampire Slayer, you might have an extra book full of vampire monsters to kill, and maybe another one explaining the minutia of the Watchers, since your players want to get really involved with them. Or, you might not need those extras.
For the moment, you don't need to worry about extra books. Start with one at most - after that, we'll see.
What do I need to start playing D&D?
First and foremost, you'll want the basic rulebook. AFAIK these are the two popular choices nowadays:
You'll also need some dice, which you can buy at game stores for a few bucks. Then you'll need at least one person to get pretty familiar with the core book, and for all the players to get a general sense of the rules. And one of you will be the game master, and that person will need to either come up with an adventure of their own, or get a ready-made adventure scenario to play out.
I won't lie - that's a lot of effort. It's particularly hard when none of you is familiar with the game - even one person who knows the game, or is very enthusiastic about learning it, can make the whole process much easier for the rest of the group.
What other options do I have?
The easiest way into roleplaying is probably to find somebody who's already in. If there's a hobby/game/collecting store anywhere near you, they might have events specially aimed at new players, or just know people in the area glad to teach you (where I live, there's a large roleplaying community, with frequent game conventions where new players are welcome to try out games). The game companies have their own programs, like Pathfinder Society and D&D Encounters, to encourage new players - you might be able to find one in your area.
You might want to try the D&D Starter Set - it's not the complete game, but it's a really great introduction, giving you the general feel of what D&D is, and guiding you through a lot of the initial confusing by fun, choose-your-own-adventure-style demonstrations. (Downsides are that (a) it's kind of expensive, and (b) if you do like it, you'd need to buy the core rulebook in addition. But it's probably one of the best introductions to D&D you can find in printed form.)
If you've got a friend who's into roleplaying, you could ask them to run a game session or two for you - just enough to give you the feel for it, and figure out if it's fun for you. Your friend might prefer some other system rather than D&D, but that's really not a problem - other systems are fine too, and having someone to ease you in is a huge help.
Lastly, you might look for roleplaying games based on your favorite movie, TV show, or genre. Those might be easier for you to get around in, since you've got enthusiasm for the setting and the style. (They also might be less confusing than D&D, just because they don't have so very, very many books to navigate around. There are lots of smaller games than D&D, this is just a pretty good way to pick a particular one.)
I hope this helps. :) If roleplaying sounds like fun to you, definitely do what you can to give it a whirl. If you can get it going, you're in for a treat.