What pen and paper system would come closest to the advancement mechanics of the computer game Diablo - in particular class advancement with a talent/ability tree?
I don't know how well it represents the game, but there is an officially licensed series of Diablo II supplements for D&D 3rd edition.
@Sebsmith's answer contains information directly from these books. Favor voting on his answer over voting on this one.
Okay, so I dug out my copy of the D&D Diablo 2 Diablerie, and it doesn't map abilities quite the way the computer did. For instance, you can't buy an ability again for better effects. Instead, it appears that it transforms abilities into D&D equivalents and splits them into six groups. At most levels you get a "magic ability" from one of the groups, and which one depends on the group level you are at. A few classes, like the Necromancer and Sorcerer get access to spell levels instead of most "magic abilities," but the spells seem game based glancing through them. The place the book best lets you emulate Diablo is with the item creation, since it has a long series of table to make rolls on, many of which have sub tables.
I'd be happy to edit this answer if you or anyone else has more specific questions about the book.
To begin, we must deconstruct Diablo:
Looking at the wiki:
- Items and Inventory
- Player Development
- Skill Allocation
- Skills and Combat
- Skill Hotbar
- Resource Systems
Now, no single game is going to give us everything. Looking at the diablo experience, we have a game and a meta-game. The game is the frenetic clicking during combat, carefully spamming your abilities to eliminate large groups of mobs. In a way, this would be best approximated by a turnless system.
Items follow a theme and variation on base item quality, then slot, then adjectives:
Magical items have a prefix, a suffix, or both. Rare items can have up to six modifiers; 3 suffixes and 3 prefixes, or any combination thereof. Magical items have a 50% chance of a suffix only, 25% chance of a prefix only, and 25% chance of both.
Giving us a "Bronze Hand-axe of Strength" for example. This is best accomplished through a card-based item system, where you have a deck of base items (sorted by "difficulty") with a prefix deck and a suffix deck. The high level of weapon simulation is not appropriate for a PnP game, and therefore we won't look too closely at reach or swing-speed, unless the opportunity presents itself.
The items and "what trees do I level" mechanic form the basis of the meta-game. While a minimalistic game can simulate the choice of trees by offering unlimited options, the abstractness of the minimalistic system prevents the player from gaming the meta-game and coming up with fun/optimal combos in their skill trees.
(This could actually be a fun game to design, almost a cross between magic and a traditional game, but I digress.)
Using Aces and Eights (their showdown suppliment) as a base for the game is a good start, as that removes the turn-taking mechanics of traditional games and replaces them with a "real-time" feel. While it's not frantic action, it's hard to simulate frantic action with any kind of mechanical complexity.
Mapping the item mechanics and the meta-game mechanics will be slightly more difficult. Items can be handled by mapping the ranged and melee weapons of the game to the iconic Diablo weapons. Here, take the trees and stats from diablo II directly, merely translating the numbers into A&8 scale.
Items can be created on 3 index card decks. The main deck is the "drops" deck, indicating the weapon, and its number of prefixes or suffixes. Draw on the prefix or suffix decks until a legal item is found.
While this will take some effort to port, the real-time chart of A&8 will go well with the feel of Diablo. The shot clock will lend a feeling of Diablo's pixel level simulation. Taking the talent trees from Diablo directly solves the problem of the meta-game, requiring only a translation schema to work within A&8's stats.
Yeah I know, but stay with me for a moment. The background/flavor is very different but I think the system fits very nicely with the requirement of having "treed" abilities.
Just take a look at some examples from the official Exalted Wiki:
However, magic (or rather, "Sorcery") does not work along the same lines (and does have certain power level issues) but for the rest I'd say the system mimics D2's skill trees quite well.
Feng Shui is probably pretty close - many treed abillities. However, many of the abillities are modern/scifi, so you might have to trim some options to make it work.