I think you are absolutely right to be concerned about integrating with this group, particularly given what you know about the backgrounds of the two players you mention.
Before I go into specifics, and would like to talk in general terms about one thing that I feel is absolutely essential at the table for any game to run smoothly with everyone enjoying themselves.
It is the responsibility of everybody at the table to ensure everybody else at the table is having fun, not just the GM
This is really, really important. Depending on the social contract in place at the table, the GM might have a varying level of control/responsibility in terms of enforcing behaviours within the group. However, every single player can and should be aware of how their behaviour enhances or worsens the enjoyment of others at the table.
This means that if the two players you mention are told that their behaviour is specifically making you or anyone else feel upset/uncomfortable and they do not change, then it is entirely their fault and you should feel no sense of being part of the problem.
Although I can kind of see where some of the other answers are coming from when they talk about the possibility that they are just 'having a laugh' or that they 'might not know any better' from experience, 99% of the time these justifications are excuses designed to shift the blame for the situation away from them onto the person feeling bad.
The player you have experienced being sexist
The behaviour you describe is openly hostile and antagonistic, particularly if he knows that calling you 'it' offends you. Assuming that he already been made aware that others consider his behaviour sexist and he has chosen not to change, it is going to be very difficult to do anything that will alter the way he acts towards you.
The player who is a 'sadist'
You do not give details of whether they behave this way towards other PCs, or if your GM thinks their behaviour is only going to target you. In either case though, I refer you back to the point I made at the start of my answer. If they are knowingly behaving in a way that upsets or offends another player at the table then this is completely unacceptable. If they defend their behaviour with 'but that's how my character would act' then there are a number of other questions on this site that deal with that particular issue.
If this behaviour is common at the table (and in some groups it should be noted that this kind of play is actively encouraged), then the only way of working out how comfortable you are with it is to play in the game and see. After this, if you decide you are not, then you can ask them to stop, and if they refuse then you have every reason to walk away.
If this behaviour is specific to you then it is more serious as the player would be using in game behaviour to punish/bully you as a player, and if they are unwilling to stop then there is little you can do except either accept it or walk away.
Other groups keen on getting a female player
I run 3 games face to face, and each of them now has at least 1 female player. This was not the case when I started the groups, and I have worked deliberately towards attracting them. There are a couple of reasons for this - I feel that the increased diversity at the table is positive to the atmosphere whilst playing, and the male-dominated nature of the hobby frustrates me as there is no reason for it and I view it as important that this is challenged wherever possible.
I would certainly not assume that eagerness for the participation of female players is creepy (although it could be), and I would strongly encourage you to explore the other groups a little more to get a better feel for them. Remember that you can walk away at any point.
So what would I do in your circumstances?
- It is not your responsibility to 'fix' the situation or the other players, and you are not part of the problem
- Talk to the GM, outlining your concerns and get a sense of their willingness to get involved. This will be much easier with the GM backing you up if issues come up at the table
- Decide how much of your own enjoyment you are willing to sacrifice in order to play in the game. It is extremely likely based on your description of the players that you are going to be put in situations that are horribly unpleasant for you. Remember that you are supposed to be doing this thing as a hobby and to have fun and ask your self whether it is really worth it
- Your shyness will make this difficult, but the best thing you can do is to communicate clearly and calmly when behaviour makes you feel uncomfortable. If you can't do this then your options once playing the game are limited to communicating through your GM, putting up with the behaviour, or walking away
- Be prepared to walk away. You mention how close you are to your GM as a friend. Be aware that the situation you describe and the possible unpleasantness at the table should you join could well affect your relationship with them. At some point you may have to walk away if its being detrimental to your friendship
- Investigate the other groups you mention, trying not to jump to the conclusion that their eagerness for your participation comes from creepy or unpleasant motivations