Unfortunately, Grappling is a largely monster-oriented tactic: size bonuses and limitations are huge when attempting to grapple. Pathfinder improved this somewhat, but it’s still true. And Improved Grab and Constrict, both very relevant to grappling, are also found much more commonly on monsters than as class features.
As a result, the best PC grapplers are not Monks (who isn’t really the go-to class for anything), but druids, transmuters, and synthesis summoners, because they can get monster-features and improved size much more easily.
Anyway, if you still intend to do it, your goal is to lock someone down, not damage. Yes, monks deal more unarmed damage, which means they do more damage in a grapple, but the damage is still very low. The only way to do relevant damage with a grapple is by getting Constrict, and that is, once again, often difficult for player characters.
Classes: We want full BAB
BAB is added to your CMB, which means it’s crucial to your grappling skills. Whatever classes you take, losing BAB should be done only when you get a lot in return. Unless you go for shapechanging, but that’s a whole different ballgame.
Barbarian and Fighter archetypes seem to be the ones most likely to supply the features you want, but other full-BAB classes may be worth looking at.
Rage is obviously very much to your benefit. So is the big HD, and the Brutal Pugilist archetype seems perfect for you: bonuses while grappling, on a big, strong class. Uncanny Dodge is nice enough, but this is a little more relevant, and Pit Fighter is way better than Trap Sense.
The Brawler rage power is kind of obvious, but the feat is so important that you likely are better off just taking it as a regular feat. Not being able to use your attack at all when you’re not Raging seems like a bad play.
Unfortunately, there are not a lot of rage powers that directly contribute to grappling. Most problematically, no way to get Constrict, Improved Grab, or an increase in your size, all of which I kind of expected to find. The best I see is Strength Surge, which is OK enough but it’s only once per encounter.
The Beast Totem options are decent enough: Claws are light weapons and usable in a grapple, and Pounce lets you move, grab, and pin someone in one turn, which is awesome even if it takes way too long to get.
But anyway, Body Bludgeon is hilarious and you should take it when you can.
3.5 Material: Goliath Barbarian
Goliaths (Races of Stone) are one of the best races you could choose for grappling, since Powerful Build makes them count as larger for combat maneuvers and stacks with any actual size increases you get.
And a Goliath Barbarian can swap Rage for Mountain Rage, where he actually becomes Large. In 3.5, a Barbarian can also trade Fast Movement (which you’ll almost certainly never use) for Pounce via the Lion Spiritual Totem alternate class feature in Complete Champion. This is all at level 1. As such, a 3.5 Barbarian 1 is a great start, and at that point you could even just switch to Fighter for bonus feats and the like.
The Unarmed Fighter archetype is the obvious choice, and it’s a solid enough one. Tough Guy and Clever Wrestler in particular.
The rest is just feats. There are a lot of useful feats here.
Don’t completely discount Monk
Note that Monk can be full-BAB for the purposes of grappling, and the first level of the class does get you a lot of what you want just in terms of the basic feats all grapplers need (i.e. Imp. Unarmed Strike and Imp. Grapple in one level). Just because you have “Monk” written on your character sheet shouldn’t mean you have to be a “monk” in-character if you don’t want to be. You can even ignore the AC bonus and just wear armor; a lot of the time armor is better anyway.
There are three ways to go about this: take 1 level, and accept the BAB loss, take 3 levels to get Maneuver Training and lose no BAB for CMB, or take 1 or more levels of the Tetori archetype, which always gets full BAB for grappling.
Losing BAB when you don’t have to doesn’t seem too worthwhile, but the regular archetype doesn’t have that much to offer you on levels 2 and 3, other than the Maneuver Training itself. Evasion’s nice if you stick with light armor, and another feat is always good, but Fast Movement and Still Mind are minor in the extreme.
Levels beyond 3rd in the regular archetype don’t really offer much for you. You’re not looking for ki powers, and they’re pretty minor anyway. As I’ve already stated, improving unarmed strike damage through Monk levels isn’t really where you want to be for this; it’s not nearly enough.
Graceful Grappler means you can take one level, get Imp. Unarmed Strike and Imp. Grapple, and end your Monk levels right there without losing any BAB towards grappling. That does make this a very compact, effective level to take; in addition to more feats than a Fighter gets, you also get Stunning Fist, better damage (minor though that is), and that Wis-to-AC if you should choose (or be forced by having equipment taken) to use it.
Stunning Pin looks fairly mediocre to me, and Evasion is only OK. Tetori 3 looks even worse than Monk 3. So you should probably only take the 1 level with this route. That, or give in and just take a bunch; this archetype actually gets Grab and Constrict, though at absurdly high levels for the potency of those effects. I don’t really recommend that; Rage and such can do more.
Dreamscarred Press: Psionic Grappler
Psionics are very good for grappling, because psionic characters can manifest expansion and grip of iron. Powers to manifest natural weapons are also common, and natural weapons work quite well in a grapple.
On top of that, Psychic Warriors get Fighter bonus feats, which are very useful, and the Brawler’s Path is quite obvious for you, though they do lose BAB. The War Mind prestige class has its own separate manifesting or can advance Psychic Warrior manifesting, and has full BAB.
Moreover, Half-giants are a psionic race and have Powerful Build, and this is Pathfinder material. That makes them a naturally excellent choice for grappling. As such, a Half-giant with levels like Barbarian 1/Psychic Warrior 4/War Mind or Monk 1/Psychic Warrior 4/War Mind seem like quite possibly the best option you have available.