We are working on a setting with low magic for Pathfinder, and we initially started out on creating lots of custom rules to go with that. However we found that the more rules we created, the more we had to change other rules. How far do you go with customizing a rules system before you end up with a completely new one?
Houserules are an essential part of the hobby, and you have my blessing.
To answer your question, you'd have to get pretty far. For a good example of how far, check out Labyrinth Lord and Mutant Future. Mutant Future is based off the Labyrinth Lord system, but changes the setting, the class system, spellcasting, and the technology levels to the point where it's absolutely unrecognizeable.
If you look back one day and realize that you've got more new parts than old parts, basically, you've created a new system.
Once the core mechanic of the game is changed, it becomes a new game. An example of a core mechanic would be using the d20 + modifiers in D&D to resolve combat; or rolling pools of d10 in World of Darkness.
Alternatively, if the game has a specific setting, but the setting is tossed out the window and replaced with one of your own, it's a new game, specifically yours.
Once you start bending or changing rules its a slippery slope. The guys that made the game put a lot of thought into this, they know what they are doing.
The only time there needs to be rule modifications is if you encounter a situation that is not covered in the rules.