Related to this question, there are some game masters who either don't know the rules or ignore them properly. They come into conflict with rules lawyers, players who know the rules well and expect them to be followed, especially in rulesy combat RPGs like D&D where they're important.
As a player, what's the best way to encourage the GM to start playing by the rules, without hurt feelings or conflict?
You should also consider that the DM's Handbook literally tells DMs to overrule/change/ignore any rules they like if it suits "the table". Different DMs and players have different playstyles and expectations from the game. I always ran DnD with a lot less attention to the rules than some. It's not that I didn't know them (I actually have a tendency to be a rules lawyer when I'm a player), it's just that I found them to be way too nit picky in a lot of scenarios. Simultaneously referencing specific modifiers while narrating something and improvising what the PCs would find in the next room got to be too much. DnD, as with any pen & paper RPG, is a completely different game depending on how and who you play it with.
I have to agree with KRyan about trying a more narrative driven game instead of DnD. Don't get me wrong, DnD is great and every edition will always have a spot on my shelves, but lately I've been having way too much fun with Fate.
Check out Fate RPG (especially Fate Core, which is on Kickstarter atm), Savage Worlds, and West End Games D6 System. Browse through "Chris's Compendium of Free Role Playing Games" and check out 1km1kt.net for literally hundreds of free pen and paper games if you are so inclined.
In my opinion as a GM who likes to skip a rule here or there if it would make more sense story-wise, I feel the most important consideration is managing expectations.
You, as a player who relies on rules to win the combat, expect these rules to hold, so that the goblin can't just get out of a flank like that. The GM expects the combat to be flashy, and that goblin was supposed to be the one that runs away screaming for his mommy, who turns out to be a giant spider that will come as reinforcement. Or maybe the GM is simply overwhelmed by all the rules, and merely forgets them at times.
What I suggest you to do is talk to the GM about rules philosopy. Tell him that you need to know which combat rules you can rely on so that you can plan ahead your moves and deal maximum amounts of damage to those goblins, which for you is an important factor in enjoying the game. Find out why your GM doesn't stick to the rules, and find a way to compromise. If all of you like rulesy combat, you can be the "rules narrator", who tells what is happening in terms of combat advatages/disadvantages when the goblins move around, as a way to assist the GM. Maybe the GM can give you a list of rules that will always hold (and where you are invited to complain if they're violated), and a list that the GM thinks is more fluid (hopefully for some good reason).
In short: Talk to your GM to let them know that you like the rules, and why you like them, and then listen to the GM as they explain why they do things differently, with the goal of finding a compromise that is acceptable to both sides.
Ultimately, I suspect that someone playing a rules-heavy system, and then ignoring the rules, is playing that system out of tradition/habit (particularly in D&D where things got much more rules-heavy after WotC took over), and they do not actually like rules-heavy systems.
You may have more luck convincing them to try more rules-light/narrativist systems. Then, at least, the system will have more options for you to do the same sort of things, and be better equipped to handle the way the DM wants to play. If you want a rules-heavy game, playing in a rules-light system may not be exactly what you want, but it’s better than being in a rules-heavy game where you’re required to follow the rules but the DM doesn’t.
You don't, you join another game
I know that sounds a bit harsh, but it if you want a rules-focused game, seek out a GM that follows the rules above everything else. There are different kinds of GMs for different kinds of games. Pick accordingly.
When I DM my 4E game, I reserve the right to change rules whenever I don't like them. We agree about what the new rule should be and move on. My players are free to do the same. We have slowly changed 4E into a more simulationist game. We are in it for the story, and this suits our play style. If you come to my table and complain about how we play the game more than a few times, I'll ask you to leave. I make this clear up front.
Also, I will lean over and tip a frowning player's die to a better role. Yeah, take that rules!
I think this comes down to the (for lack of a better term) "social contract". Personally, I tend to house rule things extensively, sometimes very extensively. But I also tend to be upfront about the fact I use house-rules and about any big changes at the start of the game. After presenting that up front, I would be inclined to politely suggest to any player that wanted to encourage me to use the RAW that they should either give my modifications a try or that this might not be the right game for them. (And if several of the players are complaining about it, I might realize that its not a good group for me to GM for and either switch roles to being a player or leave the group on good terms.)
But I think the fact I'm up front about it is crucial. If a GM starts making all kinds of changes without that warning, especially if they change rules you were relying on when you made your character, then that is awkward. If they start doing that, then it might be time to explain the situation. Personally, I still wouldn't encourage them to use RAW, but I would encourage them to be up-front about changes and perhaps to let you make adjustments to compensate. I don't think house-rules are a bad thing, but I do think rules you aren't expecting are a bad thing.
The only time I would really encourage a GM to use RAW is if your group as a whole explicitly decides that's how you want to play and then he doesn't stick to it. At that point, he is violating the social contract and you legitimately have something to (politely) call him out on.
Be assertive and try to find out why the GM does not use the rule in question. If it is not that important/urgent, you could wait till you can talk to them outside of gaming session. Something like As far as I know, the rules as written handle X differently. Any reasons why you changed it?.
I think the scope of the problem is quite important. As a GM I end to keep it to the rules unless I explicitly state otherwise for one key reason: When the players have their copies of the books, I want them to be able to research their abilities, expected attitudes, and just plain and simple how to play.
Now, you could approach your DM/GM and ask them to be consistent, however that's about the best you can do in most situations. "My house, my rules" is the final arbiter. Should something come up that conflicts with your character build (most commonly "I can go for a month without food, water or sleep but the GM always assumes we have supplies in town", or "I get extra range on this weapon, but the GM uses abstract combat") then you may want to ask permission to rebuild your character to be more appropriate to the rules in affect.
As a tertiary note, sometimes the GM is not aware that they are ignoring or changing rules. Plenty of games I've been new to running have resulted in quick core book references followed by "let's do this for now" or "You want to do X, so off the bat here's how it'll go" and then after doing more reading between sessions finding out exactly how the core handles it. This treads into extremely difficult territory, because even if you try to be helpful it may come off as rules lawyering which most GM's I know will overrule almost without fail. There have been tables where something has come up and the player in question would say "I was looking at the rules on pg.xx but it's your game" (more respectfully sounding than it looks written). Most often, that approach leads to it being fixed in the future and the GM not wanting to leave their carefully crafted game over one little line of text.
As a GM, I guess the real question here is "why are they changing the rules?" You gave more than one reason, and how to deal with them varies depending on which one it is.
It's entirely possible there are good, logical reasons for these changes and they just aren't being communicated. You could try asking. In my game, I have a document (also on the Facebook page) that lists these modified rules so that everybody can find them, and most have a sentence or two that explains why I did that. An example of that is I removed level loss for resurrection because I WANT people to keep playing the same character if they want to as it's better for the story. Unfortunately some of these (like that one) don't come up until the situation arises, as it's just impossible for humans to remember every rule they want to modify to suit the game beforehand. :)
So in that situation, I'd just ask "why?" Maybe it makes perfect sense, and just hasn't been communicated well.
The last possible situation is that the rules are just changing arbitrarily... and if that's the case and it really bugs you, sadly the only answer may be to leave the game. In my experience a GM that changes rules willy nilly without consistency is going to keep doing so because they like that, and there isn't much you can do to stop it.
The point is that the GM is not your opponent, you are not trying to win the GM. So if he ignores a rule , he dislikes it or he has some other reason to drop it. The only case where I would confront a GM about not using some rule, is when the character would assume that "it should work that way". ie. if a magic user's spells work differently suddenly or a fighter can't hit a sure shot. The character would be confused that the "world" suddenly works differently, I would also be ok to think that "this worked this way the last time". The GM might explain that something has changed or "you'll see" or "yes, I know."
If however the GM is acting as your opponent , I would switch GM's or games. If you want to play games with opponents, play battletech. The objective is to have fun, GM as adversary is not fun. In short term it might be for the GM , but in the long term it will not be.
Sometimes, GMs do this out of fear that the system is not balanced and that using the rules as written will result in a "broken" game. Often this sentiment stems from a misunderstanding about the nature of the system. Some games are balanced at a higher "power level" than others; something like instant teleporting or scrying might devastate one game but might be a cornerstone of another.
If this is the situation, I would suggest offering to run a module for the group yourself. Play it 100% RAW, and let the GM see that the game isn't broken at all, but actually a lot more enjoyable when played as it was designed. That way, you don't have to say "you're doing it wrong" :)
Edit: I seem to not have expressed myself clearly. Obviously a lot of games do have balance issues, and a good GM can smooth out the wrinkles. But if the GM is so afraid of supposed balance issues that he never plays the game RAW, how can he know where the wrinkles actually are?
My current group is familiar with DnD but are playing nWoD at the moment. Initially we were flabbergasted and confused by how rolls were supposed to work. We kept asking "What attribute goes with [skill]?" and not finding rules text specifying it. Eventually we realized that's the wrong question entirely; the right one appears to be "What attribute is appropriate for this situation?". You can't play nWoD like you play DnD, you have to play it like nWoD and adjust from there. Balance issues are the same sort of thing. We'll eventually have house rules for this system, but the old ones we had for DnD obviously don't, and can't, apply.
