Homebrew terms have homebrew meanings
As far as I have found (I'm still looking) 5e does not use the exact term "damage modifier(s)". As such, there isn't a feature I can go off of for giving you a definition. The game defaults to standard English so I would assume it means "anything that modifies damage" but what that does or doesn't include, without an in-game example or definition, is going to be up to the homebrew's designer.
If you want to convey a specific meaning, it would probably be easiest and best to be explicit and clear about what you're trying to convey rather than bog yourself down trying to match the terminology and style that 5e already uses.
The rules do use the word "modifiers", but inconsistently
In the sections on "Damage Rolls" "Critical Hits" and "Damage Resistance and Vulnerability" (emphasis mine):
[...] You roll the damage die or dice, add any modifiers, and apply the damage to your target. [...]
[...] When attacking with a weapon, you add your ability modifier--the same modifier used for the attack roll--to the damage. A spell tells you which dice to roll for damage and whether to add any modifiers. [...]
When you score a critical hit, you get to roll extra dice for the attack's damage against the target. Roll all of the attack's damage dice twice and add them together. Then add any relevant modifiers as normal. To speed up play, you can roll all the damage dice at once.
For example, if you score a critical hit with a dagger, roll 2d4 for the damage, rather than 1d4, and then add your relevant ability modifier.
[...] Resistance and then vulnerability are applied after all other modifiers to damage. [...]
In the very first use, the word modifier is being used, presumably to mean "things that are not dice". However, the bit about adding modifiers to damage rolls soon after it is referring to the ability modifier you might be adding.
The critical hits section seems to say that modifiers are things that are not dice, since they are unaffected by critical hits which only affect dice.
And then the resistance/vulnerability section seems to be using it to mean anything that changes the amount of damage received, regardless of whether or not these are from dice.
The rules do not use the term in a perfectly consistent way so I do not believe it is a game-defined term. There are ability modifiers which are seemingly sometimes just called "modifiers" which may add to this confusion.
When writing your homebrew, I suggest simply explaining exactly what you do mean in more words than just "modifiers". If it should impact only the ability modifiers being added to the damage rolls, state this; if it should impact features like Divine Smite and Sneak Attack, state this.
