I'm playing a computer RPG, Planescape: Torment, and am enjoying it. Torment is obviously using RPG rules to determine how big of a hit I'll take, or how much damage I'll do with every swing of my battle axe or whatever, but when the game says that my THAC0 has increased or my armor class has decreased, it means nothing to me since I have never played D&D. I don't know the terminology. A manual would help, but I don't have one. Is there a primer somewhere? What game is this based on exactly, and how can I learn more about it?
Planescape: Torment is based on Advanced Dungeons & Dragons Second Edition from the 1990s. THAC0, for example, only exists in that edition and a couple contemporary D&D editions (very late AD&D 1e and Basic supplements) - it's not a term found in other games. Armor Class doesn't work the same across D&D editions, let alone other tabletop RPGs. They explain most of these terms and mechanics on the Torment Wiki, however, especially the game terminology page (omits THAC0 for some reason, but it's a character's needed roll on d20 to hit armor class zero).
AD&D Second Edition has recently been reprinted by Wizards of the Coast, in case you want to obtain a physical copy. It is not, however, open licensed, so you can't (legally) obtain the rules for free on the Interwebs.
THAC0 stands for "To Hit Armor Class 0" which means how much you have to roll on d20 to hit target with armour class 0. So as your character getting more and more experienced in chopping things with his battle axe, his THAC0 is getitng lower. If i remember, THAC0 depends on character level and specialization in weapon.
Second thing which also worth to mention is, in AD&D (Planescape:Torment is based on AD&D). Armour class can be in range <10,-10> when 10 is person without any armour, 0 is knight in fullplated armour with shield, and -10 is some guy with super duper magical armour.
This number shows how tough you are to whack with a melee weapon (like a pointed poniard or a big spiky club) or pincushion you with arrows. The lower your armor class (usually abbreviated as simply AC) number, the more difficult you are to hit. This even extends into negative numbers—so a character wearing, for example, Lower Plane Vermin Armor, with an armor class of, say –6, could walk through a scythe-slinging mob without breaking a sweat. On the other hand, another character sporting a worn cloth tunic, for an armor class of 10, would be cut to ribbons by the same attackers. The benefits of having a low AC apply to your party members and they apply to your enemies. If an opponent you’re squared off against has a thick carapace or burly-looking metal armor then they’re going to be more difficult for you to hit.
There is nothing in the manual about THAC0 at all, though. I'd recommend taking a look at the manual, as it also details the effects that your Attributes have that aren't readily apparent. For example, a high Intelligence gives you more dialogue options and a high Wisdom gives you a better chance to recover lost memories.
THAC0 and AC are terms very specific to some version of D&D. AC means Armour Class and lower means less probabilities to hit.
THAC0 means To Hit AC 0, that is the number needed to be rolled in a d20 by the attacker to hit a defender with AC 0. If the defender had bigger or smaller AC the number needed to hit would be adjusted accordingly.
Answering your more general question, about understanding RPG terms and maths, those are all explained on their manuals. There are many, many RPGs and each one has their own system (that is, their own maths) behind, usually rolling dice and comparing to some values (abilities or attribute rates, difficulties,...). Of course, those system are well explained on their own manuals.
There are a lot of RPGs available for free on the net. You just have to pick some and read the rules, if you want a general perspective of how RPG systems work. If you are more interested on the specifics to D&Ds, I think you can also find some free material on Internet.