You say you can't bring you voice down. That's fine, you don't want to strain your voice anyway.
Have you considered wearing a large, bushy (and very obviously fake) beard/goatee to the table? It would probably help people remember your character's gender, since they would see it every time they looked at you, and getting a few laughs in would also help to ease tensions about the whole debacle.
The other answers are all good, too. Upvotes all around. But personally, I try not to pass up the opportunity for a good visual cue.
I was asked for an example of something similar I did myself, so here we go. Now, this wasn't actually D&D, mind you, and not specifically regarding the issue of gender differentiation, but it's along the same lines of reminding the players who this character is without constantly having to remind them of details that are not inherently obvious by looking at me as a player (or GM, in this case):
I was running a game of Wraith: the Oblivion where I had one NPC who was particularly relevant to the plot. He knew a lot about plot-relevant things and had a fair bit of power at his fingertips, but he was also very nerdy and nervous socially - not the sort of character I wanted the players to take too seriously in conversation, otherwise they might advance through the story faster than I wanted them to.
So, I got a pair of cheap plastic glasses from the dollar store (some ridiculous shade of neon green if I recall), broke them in half along the bridge and wrapped it up with masking tape. I popped out the lenses too. Every time the players went to see this character, I would put on the glasses and start wringing my hands and making "nerd noises" while he spoke. The glasses kept slipping off my nose, too, because of the weakened bridge, which led to me constantly pushing them up - which not only completed the character for them, but also helped me stay in character, too.
It worked beautifully to set the atmosphere, it made the players laugh, and it achieved my personal GM goal of making them not take the character too seriously in role-play. It was one of the better NPCs I've ever run, if I may say so myself.
I know we all like to approach these games seriously and with respect to the rules and so on, but we're also here to have fun, and a little comic relief can go a long way to keeping everyone in line and making characters memorable as entities beyond the player who controls them at the table. In situations like this, where you want to remind the other players of a subtle detail that isn't obvious from looking at you (as a player), a good visual cue can go a long way.
If you're worried about the comedic angle derailing the game, consider something else, less funny, but equally cheap to obtain. The fake beard is what I would go with in this situation, but you do you. Just something physical and visual that the others will see and remind them you're playing a male character.