I'm thinking of making a house rule for Pathfinder which lets you counter attack when your opponent rolls a natural 1 to attack. First, is there already a counter attack system in Pathfinder? If not, how will my house rule impact the balance of the game?
There is no "counter attack" system in Pathfinder as such. First off attacks are presumed to be happening at the same time, and swords, maces, and fists are expected to be flying all the time. The closest thing in Pathfinder is the Attack of Opportunity which models how attacking a character when they have effectively left themselves open, by casting a spell, running past, or some other mechanism.
Let's look at what your suggesting as an attack of opportunity. If you only have it occur on a natural 1, it will happen 5% of the time to all characters regardless of skill. Obviously it benefits characters in melee more as an opportunity to damage and causes a problem as they are taking more damage. It's not going to effect casters or ranged characters much since they aren't making attacks in melee often. It also benefits melee characters who have Combat Reflexes (can make more than one Attack of Opportunity) as they would be able to take advantage of it more often. It may also discourage a character from making all their attacks if they have multiple attacks. If you have 3 attacks, and the lowest only has a 10% chance to hit, but also involves a 5% chance of a counterattack you might not take that chance.
In short you will increase the lethality of melee combat slightly across the board, and improve the attractiveness of Combat Reflexes, while slightly decreasing the attractiveness of multiple attacks. It probably wouldn't be game breaking, but it would certainly affect play.
It would also affect the flow of play. Your combats will be interrupted more often with players making out of turn actions more often 5% of the turn for every attack made.
One way to include this as an idea, would be to draw on the Parry class skill that was included in the Neverwinter Nights games.
It was a Dex based skill available to the non-arcane casters(except bards so only sorcerer and wizard didn't get it as a class skill). The check used a fluid DC
The difficulty class is the modified attack roll of the incoming blow. A successful parry means that the attack does not damage the parrying character. A character may only parry a number of attacks per round equal to the number of attacks per round available to the character
In addition, if a Parry exceeded the attack by more than 10 points, then it allowed a counter-attack. Each successive counter-attack took a cumulative -5 penalty to the roll(along the lines of the penalties to BAB for multiple attacks). Wearing armor applies the check penalty as per normal for Dex skills.
The big issue with just making it a natural 1 is that this makes it more frequent than critical hits, which become a problem when the players make so many more rolls than the enemies.
One thing you could do, though, is invert the threat/confirm system: a natural 1 on an attack roll creates an "opening threat," and the player must make a second "confirmation" roll with the same modifiers. If that roll also misses, then the player has provoked an attack of opportunity from the target. The target must still obey the usual AoO rules when countering -for example, they must be able to actually reach the attacker- or else the counter is negated.
This should keep things relatively balanced versus critical hits. It gives some advantage to reach, range, and higher BABs: the former two reduce the amount of foes that can counterattack you, and the latter reduces the number of times you leave yourself open in this way. Then again, that could be considered a touch of realism.