You didn't do a Session Zero - in other words, you didn't discuss and agree on ground rules for the game.
I feel you. I'm the person who makes it clear in every session zero I've ever had that I don't do PvP and I don't want PvP done to me, and if anyone disagrees the group can play without me.
I've just had a situation where we hadn't had a session zero and another player started some light PvP on me (shoving, pushing). I discussed it afterwards, out-of-character, making it very clear that this is me, not my character. In fact, my character could've easily retaliated, and in-character probably would have, but me, out-of-character didn't want to do that.
This is the kind of conversation you need to have. Make it clear that it wasn't that particular attack that day, but the whole thing. Make it clear that that's a dealbreaker and you don't want it and the choice is to stop it or play without you.
Also make it clear that this is not an in-game question. It is not about which character would win or about your character shying away from conflict. Make it clear that you do not want to engage in PvP, even if you had a guaranteed win. It just isn't the game you want to play. Or, if your position is less strict, that you don't want to do it to the degree he does. Mention what level is fine with you and where the line is.
It is important to seperate in-character and out-of-character. I believe the answers and comments that tell you to take it in-game are missing the point that this is not about you being afraid to lose a conflict, but about you not wanting to play out that conflict, no matter how it ends. This must be made clear to the other player as well. Or else you drop the game, because why should you do something that you don't enjoy?
Always remember that you can't change other people. Never forget that it is your choice to be there, or not.