I think you're always OK to say "Hey, shouldn't it be X instead?" No one gets offended at this unless the person is coming off like a know-it all (e.g. "Well I rolled this instead because per the book that's what it is nyah"). Maybe it's a mistake, maybe it's a specific thing to that encounter, maybe it's a house rule, maybe you're wrong, whatever. If you care, it's worth asking (note "asking," not "telling.")
Where it turns into rules lawyering is when you state it and the GM states their ruling and you continue to argue it. Take that off line till after the session. And yes, I'm old school, and what the GM says goes. They're running the world and the NPCs and the gods and the rules, and you run your character inside how their world works (assuming you're playing a trad RPG, of course something more shared-narrative works differently). Bringing it up isn't annoying, insisting/arguing is. Respect your GM; they are doing a large amount of work so that you can enjoy yourself. Run your own game if you want the rules to always work the way you think they should.
In my group (in all the groups I've ever played in, actually) you're always welcome to say "I think per the book I should be rolling X instead." But if the GM says "I am ruling it is Y," that's when you shut up and keep playing.
Here's a real play example, I was GMing D&D 3.5e (Pathfinder actually) and a PC wanted his snake pet to attack a guy on horseback. I said "No, it can't reach." He said, "Well technically there's nothing in the rules that says it can't." Which is true, the default D&D combat rules don't take 3D into account worth a darn. I said "Yes, that's true. But it can't reach. Next!" Being a good player, he didn't argue the point as I had ruled and it was clear I was putting game world realism over some arbitrary ruleset. The snake had a climb speed so he commanded it to climb a tree and drop down on the guy, which worked well and was clever.
What can help sometimes as well is asking about different approaches. If it really is a "roll Dex vs roll Str" question, you can help by justifying it in game - "Well instead of an agility trick to distract him, I want to use my massive strength to break the beam he's on and throw him off." (That's a real play example from a Savage Worlds game).