In order to add customization to my game, I'd like to add new rules that change certain traits or strengths from different races and classes. Is this allowed or not?
Communication is the key
D&D homebrew is the means by which you create your own environment, monsters, rules etc., so it’s entirely acceptable, if the changes are clearly communicated before play.
If I were the DM, I would make sure everyone agreed with the changes and everything was clearly understood. I would take time to address everyone’s questions.
I would also limit these changes to a player group I have played with before. If there is a newcomer, I think that adds additional dynamics that would add additional responsibilities, thus making DMing more challenging.
I would ask why you are wanting to change traits and strengths. This could create an imbalance in the characters. It could have unintended side effects down the road. I would be hesitant to change too much. If some unintended consequence is discovered later during gameplay, there is nothing to fall back on to help resolve the issue.
If it was me, I would tread lightly when making such changes.
The limit is what your players will accept
The concept of Rule 0 is that the DM has the freedom to override and change all of the rules. The GMs freedom to change things as needed and make rulings for their own game is mentioned in the rules so, arguably, even Rules-as-Written the GM has no limits.
More than that, with the exception of some organized play, there are no rules police and nothing to enforce the rules. As long as everyone in the group is on the same page, nothing stops you from changing every rule in the book so long as your players don't revolt and abandon the game entirely.
The more interesting question is how much should a GM change things and how should the GM go about doing it. A GM, especially a novice GM, is likely well served by following a few guidelines:
- Be up front with your players. Sometimes of course you need to make a rulings call on an edge case when it comes up. But if you are planning ahead of time to make a genuine change, you should explicitly inform the players as early as possible, especially if this change might affect how they build their characters.
- Remember DnD is (normally) well tested. This doesn't always apply to games from smaller publishing houses, but first-party material for DnD normally goes through a fair bit of testing for balance and playability. It isn't perfect, and "broken" things definitely gets published, but more likely than not the designers considered the balance effects more than you have. When making a significant change, you are changing that balance. In other words, be aware that your change may have unintended consequences, especially if you care about balance.
- If you are dealing with experienced players, significant changes will run against their expectations. This isn't always a bad thing, but its worth remembering that you may be increasing the cognitive load on your players if they are used to playing without homebrew.