For the first time I have been invited in a group where I don't know almost anybody (it's me and a friend of mine and random people we met in a game association). Since I have social anxiety I am pretty worried at the idea. I am not great at playing since I haven't played many times and I don't know the rules (they play D&D 5 and I used to play Pathfinder). What are some social rules or behaviors I have to keep in mind?
The important things to keep in mind
Be yourself! They invited you because they think you'll be a boon to their group, don't be someone you're not, that's not who they invited.
Be nice! Obviously this is like fitting in 101, but, generally, people treat you the way you treat them. If you're nice, they'll be nice back.
Be considerate. If You're stepping into a group that's already a thing, and they're playing their game. Don't be a rules lawyer, stop and learn their table rules, how they play etc. If you're stepping into a new group with folks you don't know (the case here), then take some time to feel folks out in the group before being super assertive.
There are a few other things you can do, make sure you know if you need a character in advance, if so, work with the DM for what the specifications are. Make sure you know what the rules of the table are and if they have a social contract (probably not a bad idea to ask before you come about this). Most groups are super accommodating of new players, so I wouldn't worry too much about that. But if you are feeling anxious, have a chat with the DM about it and make sure he understands the anxiety and that you haven't played this system before (shouldn't be surprising it's brand spanking new).
The last thing you might want to do is go ahead and grab a free copy of the BD&D 5e rules and take a look. It's still D&D, but there are substantial changes from PF that you might want to get a look at. Wizards hosts this free on their website.
When in Rome, do as the Romans do.
It's usually accepted that new members of ANY group (professional, social, gaming, etc) will be a bit quieter and reserved at first, so take that time and learn the ins-and-outs of the group. Some groups are boisterous and jovial, some groups are hard-core tacticians, some groups are in-character-roleplay all the time. Give yourself some time to figure-out how the social dynamic works.
Here's a couple other tips regarding new groups:
Talk with your group to clarify what your PC's role in the group will be (healer, Damage, etc). Do that part.
Be helpful whenever you see an opportunity, but don't let other players boss you around.
Pipe-up when you feel you need to, but make sure to let other folks get their time in the spotlight.
Ask if there are any important house-rules you should know about.
Lastly, after a few sessions, if you're not having any fun, it's OK if you need to quit. Be up-front about it and tell everyone with as much honesty and notice before the next session.
First Relax; DnD 5 hasn't been out that long and there is a very good chance that the others will still be learning the ropes.
Second Be Prepared; If you can down load the Free basic player rules from Wizards of the Coast. And look through the rules, if you can't no problem. Come with note paper, pencil&pen, dice.. You don't have to come in with a trunk but be ready to get to work creating the character.
Third Cooperate; The DM may have an idea of how he wants his campaign area to work and what kind of quests he wants the party to deal with, but may also want an idea what motivates your character. So be ready to accept or offer suggestions if needed by the DM.
As a fellow player in the same boat my advice is before you reach the table know what you are looking forward to in your character concept. A new group who has some older members will have a bit of method to their madness, and you need to know what you want out of this game so you don't accidentally just become an extension of their ideas.
But above all be yourself, listen as much or as you talk, and be flexible so your play style compliments the other players.