I have a low level party coming up on a boss fight and for he style of the campaign there is a certain demon that works really well as the boss. The problem is the CR for this creature is far too high for the party. Is there a template or rule system for decreasing monster power much like monster advancement rules but reversed?
There are a couple of negative CR templates (e.g. incarnate construct), but none of them apply to demons to my knowledge.
Ultimately, even if such rules existed, they wouldn’t work well, which is probably why they don’t exist. The issue is that while you can advance a creature (at least a couple levels) by just increasing numbers, and you can always staple more abilities on (e.g. with templates), simply decreasing numbers will not always work for de-powering creatures because they have special, non-numerical abilities that are problems for lower-level adventurers. The only reason stuff like incarnate construct kind of works is because it strips a construct of all the myriad excellent construct traits, i.e. the template knows of some abilities it can remove to reduce CR (and even so, it’s still eminently abusable).
So really you have to reduce CR manually, on a case-by-case basis. We can probably help with that, you know, if you want to ask another question about your particular case.
While not a template, applying an arbitrary number of negative levels and calling it a day is a quick and dirty method of reducing the effective CR and ECL of a creature.
For example: grant 2-4 negative levels to a given creature and call that a template, "Wounded (X)" or perhaps "Weakened (X)", where (X) is the creature in question.
However, as CR and ECL are NOT the same thing, how to recalculate the CR based on negative levels is a bit of a mystery not answered in the rules. Also, please keep in mind that all CRs (and ECLs) are estimates, and can be influenced by terrain, weather, allies, and strategic planning on the part of the creature/monster.
Sure, but it's a bad idea
...if taken too far.
You can level a monster down by HD, just like you advance it. I've done it and it works okay. The problem is that many of the monster abilities turn it into a glass cannon: its abilities are much more likely to kill the PCs, even though HP drop enough that they can kill it quickly too.
(Special mention goes to a de-leveled purple worm — Swallow Whole stays very nasty at lower levels.)
If you have an ideal "boss" in mind, and it is more than one or two CR too high, instead either try building a new monster along similar lines at the correct CR or re-fluffing a correct-CR monster to match your ideal (re-fluffing is a versatile and good solution to many problems).