Yes, magic to compel suicide or homicide are available in D&D.
Homicide isn’t even particularly rare—since most creatures in D&D are at least circumstantially willing to use violence to solve problems, getting one to choose a new target they wouldn’t otherwise consider an enemy often isn’t very difficult. Many forms of mind-control allow targets to resist orders that go against their very nature, but killing isn’t, as a general rule, against the nature of most characters in the world of D&D (though it is, of course, possible that a given situation will be one that the Dungeon Master rules resistable, either because the character actually does oppose killing or because of something about this killing).
As for suicide, most forms of mind control do not allow “Obviously self-destructive orders” as D&D 3.5e’s dominate person puts it, but more powerful and/or more specific options exist that can, in fact, compel suicide. For example, death urge causes a fairly powerful—albeit brief—suicidal urge that can certainly cause death. (Note that death urge isn’t available to the “wizard” class, per se, but the distinction between a wizard and a psion gets into semantic details about how the supernatural works under D&D’s particular rules, which doesn’t seem relevant to the question.)
Out of curiosity I wondered just how powerful a wizard might become and whether they could compel a fellow PC to commit murder or even suicide. Could the GM intervene?
Under the rules, yes and yes. Many—if not most—editions of D&D don’t make any firm distinction between player characters and non-player characters, so anything PCs can do to enemies, they can also do to each other, or have done to them by enemies. As for GM intervention, “Rule 0” of the game is that the DM can always intervene, including by changing the actual rules, in the name of improving the game—which is easily plausible for a case like this, if most of the group isn’t comfortable with this action, as seems likely to be true at most tables. However, absent an invocation of that Rule 0 authority, the GM may or may not really have a chance to intervene with this: dominate person, for example, calls for the DM to make a ruling on whether or not the particular kill order goes against the subject’s nature (in which case they could attempt to resist). For death urge, it’s automatically assumed that the subject is going to resist. But in both cases, if they fail to resist, then per the rules, the subject is going to perform those actions. Anything the DM does at that point to prevent it is changing the rules—which means they’re using Rule 0.
I make the distinction on whether or not the DM uses Rule 0 not because using Rule 0 is in any way illegitimate or inappropriate—it’s precisely the tool the DM is supposed to use to avoid uncomfortable-for-the-players situations like this. I do it merely to indicate, in a sense, the “default” expectation of the game is, which is that dominate person and death urge are things that can happen to people and can cause involuntary homicidal or suicidal behavior.
Finally, I would offer a commentary here: many, many tables, probably most, ban “player-vs-player” behavior, either explicitly, or implicitly as part of a “gentlemen’s contract.” It simply isn’t a part of what many people expect or are looking for when playing Dungeons & Dragons, nor is it what the game is really about (in other systems, that may well not be true). Furthermore, death urge is a fairly obscure, little-known power, and dominate person is something that a lot of tables are kind of uncomfortable with to begin with. So I would say that it would be fairly rare for this kind of thing to actually come up at most D&D tables. For instance, I don’t believe death urge has ever been used at a table I’ve been at, and while dominate person has been, it’s been quite rare and the usages have generally been far more mundane.
