Creatures that threaten a foe aren't normally allowed to make an attack of opportunity against that foe just because that foe makes an attack with a reach weapon against a nonadjacent creature
An attacker that's armed with a typical melee weapon that possesses the quality reach and who makes an attack against a foe that's more than 5 ft. away does not provoke attacks of opportunity from foes that threaten that attacker. Table: Actions in Combat says what does and doesn't provoke attacks of opportunity, and a standard melee attack typically just doesn't provoke attacks of opportunity, even if the attacker makes an attack against a nonadjacent foe with a reach weapon while the attacker's threatened by a nearby foe.
However, on Cover, in part, says, "When making a melee attack against a target that isn’t adjacent to you (such as with a reach weapon), use the rules for determining cover from ranged attacks." Those rules say
To determine whether your target has cover from your ranged attack [or a reach weapon attack not against an adjacent foe], choose a corner of your square. If any line from this corner to any corner of the target’s square passes through a square or border that blocks line of effect or provides cover, or through a square occupied by a creature, the target has cover (+4 to AC).
Thus surrounding a typical reach-weapon-wielding creature not only impairs the creature's normal attacks against nonadjacent foes by giving those nonadjacent foes cover but also prevents the reach-weapon-wielding creature from making attacks of opportunity against those nonadjacent foes ("You can’t execute an attack of opportunity against an opponent with cover relative to you"). Both of those are a big deal, and together they effectively negate any benefit of a typical reach weapon. (And, of course, if combat is taking place from fewer than all sides, surrounding per se isn't even necessary!)
So, to be clear, the designers did recognize the similarity between making an attack with a reach weapon against a nonadjacent target and making an attack with a ranged weapon, and the designers did build that similarity into the game's engine. What they didn't do is make it provoke and, instead, simply made it more difficult when folks are in the way. I can't know why they didn't make it provoke, but here's some idle and utterly unproven speculation:
- Were an attack with a reach weapon against a nonadjacent creature to provoke attacks of opportunity from foes threatening the attacker, very few folks would use reach weapons. The disadvantages of using a reach weapon are already fairly significant, what with most folks not being able to threaten adjacent squares while using one. Adding another limitation sees reach weapon use dwindle even more when they're supposed to be a fairly common thing.
- Were an attack with a reach weapon against a nonadjacent creature to provoke attacks of opportunity from foes threatening the attacker, attacks of opportunity would be even more confusing. Currently, melee attacks typically don't provoke attacks of opportunity, but this rule would add a rather large niche in which melee attacks would provoke attacks of opportunity. Already among the game's most complicated rules, making the attacks of opportunity rules even more complicated would alienate even more potential players.
(Undoubtedly, others can imagine other possible reasons!) Anyway, if a house rule that says A creature that makes an attack with a reach weapon against a nonadjacent foe provokes an attack of opportunity from each foe that threatens that attacker feels right for your campaign, talk over the house rule with your group, make it a thing for your campaign, play with the rule for a while, and see if it has the results you want.
It's possible that the result will be Folks love the new reach weapon rules despite the danger, and they keep using reach weapons even if they die doing so! then your house rule's probably perfect, and the designers probably should've gone that route themselves.
However, I suspect the result will be No one uses reach weapons anymore because everyone says they're just not worth it. Combat becomes slightly more boring, and monsters with reach—because they aren't using reach weapons—become that much more dangerous. Nonetheless, this might be exactly the result you're after if your campaign was previously nothing but reach-weapon wielders (e.g. every PC used a reach weapon, and the players viewed any magical weapon that wasn't a magical reach weapon as vendor trash). Sincerely, if you give that house rule a try, I hope everything works out the way you want.
