Betrayal is achieved through imperfect information, possibly conflicting goals, and the ability for orders to be miscommunicated.
(Caution, game theory ahead)
I'm going to assume that you're familiar with the Prisoner's Dilemma, the iterated prisoner's dilemma, the stag hunt, (Kuhn 2009) and the problems with resource availability on evolutionary fitnessa.
Betrayal is actually an interesting question to model in game theoretic terms. (Akiyama & Kaneko 2000) b. Most environments tend to have huge amounts of resources, or few amounts of resources. Huge-resourced environments tend to reward cooperation. There's lots of stuff there, there's no real problem with the other guy having stuff and by working together, both people get more stuff. c
Scarce resourced environments start looking very much like the cold equations. There is no room for cooperation. If there's just enough resources for one thing, you... fight over the resources and hope like hell the environment changes before you need to reproduce.
Neither of these are interesting.
Evolutionary pressures on betrayal
Competition produces fascinating evolutionary pressures when there's enough resources to not worry too much about simple reproduction, but not enough resources to just go somewhere else when someone else is better at it than you are (Day 2000). Replace "reproduction" with activity of choice here. d
Now, this specific problem has actually been well covered in game theory. Consider the Centipede Game (Rosenthal 1982, McKinley & Palfrey 1992) and why there are some interesting utility calculation problems there (Berg et. al. 1995).
But we can use the centipede game, assuming we have accurately modeled utilites (and we can, because we can declare a set of resources as utilities and provide weightings for the players) as the basis for determining when defection should be possible.
Specifically, if we look at the centipede game, we can see that we make it interesting by adding a "minimum win threshold" (i.e. you lose if you have less than 40 units of utility, period) and making it so that the period between "Being able to win" and "both players win" is quite long.
Of course, both players here exist in a state of perfect information with the other. They both know that being able to honor at every step means they both walk away with 100 units (to take the specific values of the video into consideration).
The Fog of War
Therefore, we need to add in "limited information." Moving to a limited information state does move us away from perfect game theory, but is still modellable. (See mixed strategies and rock paper scissors) (Smith 1993). The way we add in limited information is by borrowing techniques from Kriegsspiel. Specifically, the information players get about their environment is imperfect, their control over their environment is imperfect, and actions have detectable precursors (Ciancarini 1997)e.
Therefore, if we set it up that cooperation is necessary for a certain duration in order to win, but no player has complete confidence in the other player's actions over a large enough time horizon to make always-cooperate a strategy, and to insure that victory is possible in the "mid game" and that resources are, only if perfectly split, available for mutual wins, we get an environment where betrayal may happen, may be seen to happen, and that perception of reality may ... or may not be acted upon.f
Therefore, players, when playing, cannot know the entire state of the board. Instead, they must only be aware of what the other players communicate, public actions, and any data (however imperfect) their intentionally-actioned spies provide.
Where this occurs in RPG games right now.
The design space of Role Playing Games invariably begs the question of what a role playing game is. Taking a broad minded view of "if they bothered to show up at GenCon or Origins, they're probably an RPG unless they identify otherwise." The best example of situational betrayal is in the LARP genre.
Specifically, the best cooperate/defect game I know of is the National Security Decision Making Game. Players play various heads of state and subdiary members set in the real world. g. Each player is assigned personal objectives at the start of the game.
Players are adjudged winners by how well both they and their country do. Thus, treason is not encouraged (usually), but making sure that you always have a finger in the pie as your country grows is one of the better routes to victory.
Therefore, to provide a forced mechanism for betrayal besides limited information, give players secret objectives. These objectives may or may not come at the expense of the other side. However, as repeated games of Homeworlds have shown me, when prohibited from revealing your personal goals, players will inevitably construe any action another player takes as sinister and a likely betrayal. h
If you set up a board with this limited information, resources that are valuable and somewhat scarce, the ability for mis-translated orders to occur (but not often...) and the ability of third parties to come in and literally eat the first two players' lunches, you'll get a space where betrayal is a valid strategy, either as a preemptive response to a formal betrayal (albiet by sacrificing more resources), or as an incorrect response to a mis-signaled betrayal, or as a forced response due to the actions of third parties. And thus, you can have situations where the rulers of a country are quite happily chatting with each other, and then some whacko decides to achieve peace on earth.
This is best achieved through committee.
If you have groups of players, with problems sufficiently complex that they have to be fragmented between the players (hello limited information and miscommunication!) that are working for not quite complimentary goals in the long term... then all it takes is one spark to send the whole world up in flames.
This provides for interesting tension and drama as communication issues are resolved, threats are seen as real threats or miscommunications, and hidden pressures that may be not seen from the other side.
a The links go from easy to hard-scholarly. This is not really the place to cover the links between evolutions on game theory, and especially not the place to link Press and Dyson's fascinating zero determinant strategy for prisoner's dilemma that outcompetes iterated tit-for-tat.
b The point that Akiyam and Kaneko bring up is people maximizing personal versus group utility. Betrayal, at the end of the day, is about one party deciding that their personal utility overweighs the positive group utility that would occur from cooperation. Beyond that, they are also right that the utility landscape changes from action to action. This is actually a remarkably complex game-theoretic problem. So I'm going to, in general, treat this part as a static game landscape.
c Group policing strategies change as a function of resource availability (Frank 1996). If we're going to extrapolate from Van Dover (2000), middling resources is a simpler way of stating brief resource blooms or varying resource availability. Lots of growth (and mutations) happen during high resource times, and then as the environment becomes more austere, the organisms that die... don't breed (Day (2000).
d See the intelligence hypothesis here to get a sense of why this is interesting.
e Technically there are domains of game theory that extend into imperfect information realms. But by this point we're getting into graduate level research. The point I'm exploring, here, is that the rational action with imperfect information may be the wrong action. And from a games perspective, that's fun.
f And this is where we celebrate the fact that Stanislav Petrov saved the world from nuclear war in 1983. He is a hero.
g My most vivid memory here was, as President of Iran, my security minister nuking me with a Russian suitcase nuke, framing the Russians, to force me to ask for American aid so that he could depose me as a western sympathiser. Well, and as a member of the Japanese Diet sitting in a nuclear bunker as the rest of the government argued over succession issues as the bombs from china fell... Players tend to let the nuclear football go to their heads. And the DPRK almost inevitably picks a scrap with ... everyone.
h See also: Shadows over Camelot. Where there may be a traitor in any given game, but time pressures and other mistake sow the seeds of doubt and accusation.
Akiyama, E., & Kaneko, K. (2000). Dynamical systems game theory and dynamics of games. Physica D: Nonlinear Phenomena, 147(3), 221-258.
Berg, J., Dickhaut, J., & McCabe, K. (1995). Trust, reciprocity, and social history. Games and economic behavior, 10(1), 122-142.
Ciancarini, P., DallaLibera, F., & Maran, F. (1997). Decision making under uncertainty: a rational approach to Kriegspiel. Advances in Computer Chess, 8, 277-298.
Day, T. (2000). Competition and the effect of spatial resource heterogeneity on evolutionary diversification. The American Naturalist, 155(6), 790-803.
Frank, S. A. (1996). Policing and group cohesion when resources vary. Animal Behaviour, 52(6), 1163-1169.
Kuhn, Steven, "Prisoner's Dilemma", The Stanford Encyclopedia of Philosophy (Spring 2009 Edition), Edward N. Zalta (ed.), URL = http://plato.stanford.edu/archives/spr2009/entries/prisoner-dilemma/.
McKelvey, R. D., & Palfrey, T. R. (1992). An experimental study of the centipede game. Econometrica: Journal of the Econometric Society, 803-836.
Rosenthal, R. (1982). "Games of perfect information, predatory pricing and the chain-store paradox" Journal of Economic Theory, 25. 92-100.
Smith, J. M. (1993). Evolution and the Theory of Games (pp. 202-215). Springer US.
Van Dover, C. (2000). The ecology of deep-sea hydrothermal vents. Princeton University Press.