To answer this, I think it best to look at 3 distinct vectors of analysis. Was it Rules sound, was it socially acceptable, and was it an appropriate choice for your character (and the rest of the party and NPCS). I’ll go in depth, but for those who don’t feel like reading the full analysis and Socratic debate, I’ll give a summary. Yes, it followed the rules. No, it wasn’t something most in the RPG community would consider socially acceptable. But Yes, it can be appropriate, so long as it is in character and other characters (PC and NPC alike) react appropriately, creating an opportunity for character development or both of you.
Now the Analysis.
First, the easiest, are there any rules explicitly forbidding it? No. Yours and the GM’s interpretation of Dominate Person seems sound to me, and there are no mechanical issues here; as far as spells are concerned there is no difference between a PC and an NPC.
Secondly, was it socially acceptable. This is shakier. Within your particular group, it evidently was as only one player complained. In the larger RPG community however, controlling another players character is generally considered a faux-pas. It detracts from their ability to play the character as they see it, and ruins their fun, especially if they aren’t given any way to get around it or adress the issue; even more so if they are given no warning ahead of time (it was unclear in the question wether attempts were made to discuss the issue with Barbarian player ahead of time).
Casting hostile spells against party members falls into the “Player vs. Player” area, and generally is going to antagonize other players; in some cases, more so than simply killing them. A PK will tick people off, but at least you can come in with a new character. Dominating another PC is effectively turning their character into your cohort (or worse, an NPC) whenever you want, and absolutely detracts from the player’s experience; frankly, if this had happened in one of my own games (DMing or playing) I would expect an immediate “What the Hell Hero?” from the dominated player (and likely the rest of the table).
Admittedly the closest approximation I’ve had was (my own character) using “Hostile Levitation” to keep the party’s AntiPaladin (Very Chaotic Game) from shredding a church. Notably in that case, I chose “Hostile Levitation” rather than something more direct because it allowed his character to still play, to still react (by shouting that the Priest of the Church of Abadar we were in was a Fascist, through the stained glass window he was floating by, but that’s beside the point). Dominating a player though, means that they can’t even complain in character, especially if they don’t know that it’s even happening or who’s doing it. Maybe this will come off hypocritical, but to me, restricting another player’s options to avoid conflict is alright (as long as it’s justified in character), but completely removing their choices or character opportunities is not, as it detracts from their fun.
Now to third branch of analysis, was it appropriate in story, given that, as presented, it seems to have been motivated more by player annoyance than character annoyance (particularly among other players). [Before stepping here, I should state my own bias towards Chaos, so if it sounds like I’m playing Protean’s (or more often Azata’s) advocate, it’s because that’s my natural tendency.]
As presented, your character was noted to be Lawful Good, and restricting other people’s options or freedoms for a greater purpose is the classic embodiement of what law is (at least from a chaotic perspective). That is the function of lawful society, rules are put in place to keep people in line. [It is worth noting here though, that depending on local edicts, what you did may be questionably lawful, if the agressive use of such spells were forbidden, or if their use were restricted to individuals of certain position or with some lawful authority over the victim, i.e. a law-enforcement officer could do so, or a lord over his vassal, but it may be a violation for a common citizen to do so to another citizen of equal peerage].
However, while what you did was, most likely, lawful, it was not “good” by the standards of a full two thirds of that spectra. And given that you described the group as generally “Good” not “Lawful Good,” that leads me to guess there were some Neutral to Chaotic Good characters (The rogue maybe?) in the party, though even a Paladin may be uncomfortable with this, depending on deity.
Perhaps the most common moral dictum is the Golden Rule, “Do unto others as you would have done unto yourself,” and this seems to fit to a Tee the Good alignment in Pathfinder. So from with that in mind, how would you feel if the tables were flipped, if in a Chaotic Party, another party member attempted to Dominate your character for whatever reason? Would you be frustrated, angry or annoyed? Then you can understand why the Barbarian player was unhappy, and more to the point, you can understand why doing this, while lawful, is not a “Good” act.
So here we get to the old to be good or to be lawful debate. Your character, as described and being a follower of Abadar (LN), I’m guessing veered toward Lawful, in which case this makes sense in character. An individual’s habits, while not evil, were causing unnecessary trouble, so you resolved the issue by placing a restriction on their freedom. You created a law [wether you had the authority to do so is another matter, but that’s for the Abadan theologists]. An instrumentalist approach, the ends justify the means.
However a Chaotic or Neutral Good Character would likely see things very differently. To a Neutral Good perspective, you have chosen to do harm to another person not because it was necessary, but because it was the most expedient solution, you could have talked to them or given them a warning but instead you chose to just put them in, metaphysical, chains. Looking at the core Neutral Good Deities, from a Sarenraen perspective you gave them no chance at redemption and from a Shelynite perspective you have forbidden their self expression, both No-No’s in their respective faiths, and while their theology may not be relvant to your character specifically, It would likely be relevant to other party members (especially if the Paladin was a follower of either [Or other more “good” LG deities, such as Erastil]).
I’m not sure it’s even worth bringing up how a Chaotic Good character or a follower of the deities of that area (Cayden Caylean, Desna) would react; the answer is poorly. Both Cayden and Desna [I say this as someone who almost always plays Caydanites or Desnans] are overwhelmingly against any form of slavery, which it isn’t hard to argue spells like “Dominate Person” fall under. You are explicitly controlling another person without their consent, not in the way Suggestion nudges them toward a course of action they consider “reasonable,” or in the way Geas can make a target pursue a goal or follow an edict, but explicit mind control. Domination. And while I am using Cayden and Desna as examples here, this extends to most character’s with a Chaotic Good mindset as well, and a large portion of Neutral Goods as well.
Taken together, we find a situation where it is questionable why the other character’s were okay with this, or why anyone who might have noticed was (given spotting someone being affected by enchantment magic isn’t hard). While it may have been in character of your character to do this (an argument you could very easily turn back around on the barbarian player), was it in character for others to react as they did and for no one to call you out on this.
So, to reiterate the summary. I do not believe that this action violated any rules in game, and I believe the DM’s ruling was justified in this scenario. However, your choice to do this constitutes a significant faux-pas without the culture around Tabletop RPGs, and the Barbarian player’s frustration was entirely justified. But, that does not mean that it was wrong of you to do it, as a character; However, the other players, and the DM through the Medium of NPCs, should have called you out on it or otherwise taken some action to turn it into an opportunity for character development for both of you.
