Ultimately, this is such a massive change that the only way to answer it would be to playtest it, and playtest it very, very thoroughly.
One thing it would not do is “force” those inclined to optimize to stop doing that and play whatever you might imagine to be a “casual” character. Constraints are part of optimizing—if you have this hard cap, then you optimize by finding the most efficient, reliable way to reach it. If over-optimization is a problem, the solution must be out of game. It’s a metagame problem, a disconnect between the expectations and desires of the optimizer and the non-optimizer.
On the other hand, changing the rules of the game can change the nature of an optimized character, and that can improve your game. The optimizer might still be optimizing, but if you set the rules right (a big if), you can make that optimal character be the sort of character you want from the system. In fact, this is, generally speaking, the entire point of mechanical balancing—to try to ensure the rules are set such that players who use the rules don’t produce problematic characters, no matter how tricksy they get. This is extremely difficult to do, and may require sacrifices in other areas of the game if you’re really determined to do it, but it can be done—by and large, D&D 4e is considered to have come pretty close, for example. But again note the massive differences between 4e and previous editions, which were necessary to accomplish that.
Anyway, as I see it, there are basically three options for the cap, and I feel I can speculate—somewhat—on how each would effect the game. I’m not going to promote any particular curve as falling into any of these three categories—that would, again, be a question for massive amounts of playtesting. But I think it’s fair to say that any given curve would fall into one of these categories, and I believe we can extrapolate some of the effects from that.
“Too High”
If your cap is high enough, it might as well not be there at all—you could imagine setting a cap higher than anyone is actually able to achieve, even through the most hardcore optimization, at which point the cap has no effect whatsoever. And it doesn’t even need to be that high; it could easily just be high enough to be higher than is practically relevant to the people actually playing.
This is the simplest case: since the cap has no meaningful effect, the game proceeds exactly as it does now.
“Too Low”
So there are basically two ways you could define too low: either that it just “feels” too low to the players in question, not doing enough damage for some arbitrary standard of damage dealing that they have in their own mind, or for a more objective definition, low enough that damage dealing is pointless and you are better off with non-damage approaches. Ultimately, Pathfinder is arguably already here.
Not because the damage potential of characters is anything like low—a typical striker character with half-decent optimization, by which I just mean picking good feats and having a good class, not some convoluted combination of tricksy mechanics, can easily deal something like 150% of expected foes’ hp in a round. In a lot of ways, if you aren’t killing things in one shot, you are below the curve for Pathfinder strikers.
But nonetheless, the damage curve is still arguably too low. And that’s because non-damage options are just that good. A striker might be able to easily overkill something in one round, but that still pales in comparison to a spellcaster who can shut down half the opposing party—if not the entire opposing party—with a single standard action. And options for doing that very much exist in Pathfinder.
So by that argument, having any damage cap at all just pushes players towards non-damage-based options for combat—and the game was already pushing them that way. On some level, it just punishes those who resist the incentives already present in the game in order to play the sort of character they prefer—not great.
“Just Right”
But let’s say you are concerned about damage, in particular, and aren’t interested in considering the competition presented by non-damage options. Let’s say, for the sake of argument, that you do a lot of playtesting, and establish a curve that delivers an engaging combat experience, where characters feel effective, without seeming “too powerful” for whatever definition you want for that. What effect does that have on the game?
The primary effect is that characters who want to deal damage now have to also do non-damage things; putting everything into damage-dealing is pointless because of the cap. That doesn’t mean you stop optimizing damage! It just means that your goal now is to get there as easily/quickly as possible, with as much left over as you can have. And then leveraging the left-over resources as much as you can for non-damage options that are compatible with your damage-dealing, which basically means things you can do with actions you aren’t using, or that you can add on top of your attacks without interrupting them.
Such characters are, again, already pretty strongly incentivized by the game rules. And the increased versatility is probably good for the game.
The real issue then, is actually determining that curve (good luck!), and then ensuring there are plenty of options for reaching the curve and then adding onto it with non-damage options so that you don’t end up centralizing characters too much in any one direction.
This is hard. This is very, very hard. Almost no game system has ever achieved it, in my experience, including those that tried very, very hard to do so. D&D 4e is probably the closest—and probably worth studying—but still definitely had clear winners (e.g. fighter, ranger, wizard) and losers (e.g. assassin, hunter, vampire).
A note about time
One clear effect of lowering people’s damage (and, presumably, balancing non-damage options around the lowered damage as well) is to make combats take longer. Consider this very carefully—even combats decided in 2-3 rounds, which is pretty typical in Pathfinder, take a long time to play. If you are doubling or tripling that, you are going to also extend the amount of game time you have to devote to a given combat. That could make the game seem to drag very slowly—in my experience, already a significant risk with Pathfinder as is.
It also changes the balance of things with ~1 minute duration, since those are no longer necessarily going to last an entire fight. That’s a big change, since “one fight” is basically what a 1-minute duration is supposed to be, and why the author of that effect chose that duration. Suddenly things that were supposed to last for a fight aren’t—and how that affects things is going to be very difficult to foresee. You either have to go through every such effect in the game, or else just have to do a lot of playtesting.
Of course, no matter what you do, you’d have to do a lot of playtesting, really, to answer any of these questions.
