RPG's are a non-exclusive clade defined mostly by either self-proclamation and/or player acclamation as a Role Playing Game.
The clade of general acceptance includes several non-unique traits:
- Role Assumption - encouragement to make decisions as if in the role assumed
- Character Scale - the fundamental unit in play is usually scaled to the individual character
- Open-ended action list - the GM/Referee/moderator can adjudicate non-standard actions.
- Lack of clear victory conditions - the players don't have a clearly defined way to "win" other than by enjoyment of the process.
- focus on story - either emergent from play or shaping play from a starting state
- character advancement - the character is able to be altered by play, usually increasing in competence.
- One character per player - usually, each player controls one and only one character.
- Campaign Play - usually, RPGs are played in multi-adventure sequences
None of these is universal in RPGs. None of these are universally absent in board games. And several games are routinely argued to be both board games and RPGs, depending upon who and how played.
Most active duty military, police and detective games have a pretty clear victory condition: Complete the mission or solve the case. Having clear victory conditions is not an exclusion from the RPG clade. Likewise, several board games lack a clear victory condition, tho' many don't consider them exactly games: Ouija, The Ungame, and several others.
Character Scale: In Mars:2100, the unit scale is actually a corporation (or a faction), and the head thereof. The majority of actions in the playtest draft are resolved by use of the corporation's attributes, not the head of that corporation. Likewise, when using Classic Traveller's Alien Module 2: K'Kree, the unit scale is a K'kree family - the dominant male, his wives, servants & bodyguards, and their wives. And many boardgames are character scale.
Open-Ended Action List: Kriegspiel, which dates back to the 19th Century, has an open-ended action list, which the referee was encouraged to adjudicate upon the merits of the situation and come to a decision on how to resolve and the outcome thereof. It's a wargame. Likewise, in the ___-world series of games, there are only mechanical resolutions for a handful of things, and anything else is either "say-yes" or assign to one of those few actions. And in Mouse Guard, the GM is actually allowed (even encouraged) to limit the ways in which GM-turn situations can be resolved. This is probably the most distinctly RPG-defining element, as almost all RPG's allow non-listed actions, even if they lack mechanical resolutions for them. It's not common in board games, but it's not absent from all board games. Kriegspiel wasn't the only one - refereed wargaming was pretty much the standard until the 1960's, and even into the 1960's remained standard in miniatures games.
Focus on Story - whether it's a story emerging from play, or intent to flesh out a plot that's already written, a focus to some degree on the story is present in all RPG's. Many boardgames, however are just as strongly story focused. Examples of story focused boardgames include Hobbit Tales, Once Upon A Time, Aye Dark Overlord, Careers, Life, and several others. In Careers and Life, like in traditional old-school RPG playstyles, the story emerges from play, in that the play of the game generates clear story events. In Hobbit Tales, Once Upon a Time, and Aye Dark Overlord, the goal of the game is to tell a story using elements on cards that are played - a sensible story using one's hand is the victory condition for OUaT and ADO.
Character Advancement - The ability for a character to be changed is non-unique to RPG's, and is extremely common in RPG rules. Again, it's not universal. Several games do not have character advancement other than by change in possessions, but clearly are meant to have emergent story, and open ended resolution is explicit, as well as self-declaration as an RPG by the author. Likewise, many boardgames do have character advancement while clearly not being RPGs, including the character scale Candamir, Talisman, and ElfQuest board games.
One to one: In general, most RPG's encourage a single character per player. This is not true of all, as mentioned with Traveller and the K'Kree, STRPG with the Binar, Star Thugs with a captain and crew, and D&D with hirelings and henchmen, and in Gygax's games, often 2-3 PC's per player.
Also, note that the clade is non-exclusive - some items defined as board games are also defined as RPGs. The classic overlap zone includes:
- Car Wars: especially in campaign play
- Battletech/Mechwarrior: since Mechwarrior's titular role is usually resolved with Battletech, and even in boardgame mode, Battletech hits most of these except focus on story and character scale
- Battlestations: the Brothers Siadek refuse to specify which, and say yes when asked "is it a board game or an RPG?" They also have released both a Roleplaying adventure and several boardgame-play focused expansions.
- The Fantasy Trip - Metagaming's magnum opus was released as an expansion of a pair of character scale wargames into a full-on RPG, but the Trademark was used for the whole line in either mode.
- Warhammer 40,000: Rogue Trader (the original one from the 1980's). Labeled as "3D Roleplay", filled with random character generation tables, a fairly strong emergent narrative expected, Character Scale (albeit multiple per player)
- Dungeons and Dragons - The term Role-Playing does not appear in the original edition of the D&D game. It was used in advertisements, and in the magazines, but doesn't actually appear in the 1976 edition of the boxed set. (Note that the 1976 is the revised version due to the Tolkien lawsuit.) It does appear in the ad copy in the back of the supplements, also dating to 1976. Many players played D&D as a form of tactical wargaming; many others as a roleplaying game, and even to present there are players playing each of the various editions as a form of character scale wargaming, despite D&D being axiomatically a role-playing game. In the case of victory conditions, several editions note that surviving the dungeon and having fun is the victory condition. For reference, the core original edition box title is: Dungeons & Dragons: Rules for Fantastic Medieval Wargames Campaigns Playable with Paper and Pencil and Miniature Figures (title case capitalization errors are from original).
Campaign Play is also not definitive - the first convention one-shots were in 1975 or so. Also, many modern RPG's are not intended for long term play, instead being one story and done. Likewise, campaign play was extremely common in miniatures games.
Overall, it's safest to say RPG's are a subset of boardgames, but that's unsatisfying to most. So...
At the end of the day, a roleplaying game is a roleplaying game because the designer thinks it is, or because the players think it is, and no clear distinctions can be made from board games. Every distinction is muddled by the overlaps.