Note: My answer was written before it was edited to emphasize "actions to avoid in a new group" rather than the "social anxiety" bit.
Rules for Dealing with Social Anxiety
- Relax. Be yourself. Some people will like you. Some will hate you. Some will find you too trivial to even do either of those things. And all of that is okay; the people who stick with you will be the ones who like you for you, and they're the ones you should be spending time with, forging relationships with, and keeping around. There's a lot to be said for being genuine.
There is no Rule 2.
At the Gaming Table
I'm going to write the below under the assumption that your first substantial interaction with these people will be at the first session of your game.
- Don't be a rules-lawyer or act like a "know-it-all" (even if you do know it all). That's one sure way to get people to dislike you instantly. It's possible that the group is much more novice than you and that you really are right while they're wrong. It doesn't matter. If you're the newcomer and the outsider, coming across as pretentious/arrogant or rocking the boat substantially is a good way to get thrown off of it.
- Examine their social contract; don't try to force your own on them. What I mean by this is that this group existed before you. They have their own jokes, their own ideas of what's funny and what's not, their own ideas of appropriateness and closeness and everything else under the sun. Their dynamic is established. It's your job to see if you can or even want to fit into that dynamic. I've showed up to many tables in the past and known in the first ten minutes that there would be no metaphorical "second date", and I'm sure they felt the same way. You have to see if you're a good fit for the group. And, if not...
- Don't be afraid to walk away. It doesn't have to be rude or confrontational. Sometimes it's just not a good fit. That's life; it applies to finding a company to work for, applying to graduate programs, finding a mentor, and everything else there is. It's important to mention because the fact that they already have an established social contract doesn't mean you have to sign it. There are plenty of other groups out there. Don't be afraid to do a little catch-and-release until you find a group of people that you really like and click with.
You say that you're "worried that [you]'ll say or do something stupid when attempting to join a group." I promise you that a good majority of the most entertaining things that happen at the gaming table are when people do and say amazingly stupid things. They always make the best stories. Just try not to take it all so seriously. Take a breath, relax, and just do you. If you do that, everything will turn out exactly as it ought.
Maybe wear some sunscreen.
Perspective
I'd actually like to elaborate on something brought up in the comments. @CameronMacFarland notes that:
"Relax" is terrible advice for dealing with anxiety. If it were that easy the anxiety wouldn't be a problem. You just need to relax! RELAX! BE RELAXED!!!!!!
In a sense, he's correct. From a certain perspective, saying to relax and just be yourself is sort of like saying, "How do you deal with anxiety? Simple! You don't be anxious." And because I think this interpretation might be easy to fall into, let me explain.
First, I'm assuming that the anxiety is in fact normal, "healthy" anxiety, not a clinical anxiety disorder. I'm not qualified to tell anyone how to deal with the latter; only a medical professional is. However with that said, I've suffered from severe clinical depression for my entire adult life and can relate to having emotions plague you which are sometimes completely outside of your ability to control. When someone says, "I'm feeling down," the response is usually something along the lines of, "Well, cheer up! Look on the bright side!" which is very analogous to the scenario that Mr. MacFarland described.
The ultimate point of my "relax" advice was this: all too often, the only thing you can control in a scenario is yourself. And a huge part of dealing with these kinds of negative emotions is adjusting your outlook. If you can't change the circumstance, you can at least try to change the way you perceive it.
Why is this all relevant?
Because by adjusting your perspective on the situation, you really can become more relaxed and more comfortable in your own skin. So what if they don't like you? So what if it's not the right fit? The world's full of more people than you could ever possibly meet in your lifetime; why waste your time on those who won't be adding value to it and making your happy?
I'm not saying it's always easy, and I'm not saying you just flick some switch in your brain and suddenly it's all better, because that's definitely not true. But the demon is inside your own head. It's in your realm. You have the power there. And you can overcome it.