The alignment system is not very good
It doesn’t make a lot of sense, there are numerous cases where the suggestions for what is in each alignment are contradictory, it relies on the poorly-explained idea that there are objective, cosmic Goods and Evils and Chaoses and Laws. It works well enough for simple adventure-fantasy where we are the Good guys, and those are the Evil guys, and everybody not really involved are Neutral, and the Dwarf is Lawful because he’s a Dwarf and the Elf is Chaotic cuz she’s an Elf. There’s no moral question that Detect Evil cannot solve.
If you want to play a more serious game where there are shades of gray, moral and ethical questions are ambiguous and difficult to answer, and sometimes people subvert various expectations of them, then the alignment system, as written, becomes largely useless. It might make for a convenient short-hand, but with all the hang-ups and arguments that it causes, I don’t consider it useful even for that.
And the reason that all of this is the case is because the different alignments are very poorly defined. Sometimes it seems like it’s just the action that is Good or Evil, regardless of why you do it, other times it seems like your intentions matter, and so on. Law and Chaos are that much worse, as indicated in my answer to What is a good way to explain the difference between a Chaotic Neutral character, and a character who is just crazed?.
So I personally would not pay any attention to the alignment system’s perspective on this at all. If you insist, I suspect this is considered solidly Neutral rather than Evil by... most of the published books, which is to say there are still quite a few that contradict that. Unless he’s doing things that obviously just hurt people and don’t even benefit himself (stealing some dude’s medicine that he needs to survive, but which does little or no good to anyone else), it’s a Chaotic act but not an Evil act by the stupid rules.
But I’d just ignore it and figure out your own story
Behavior like this has traditionally been taken very dimly. He could (and probably would, were this some actual reality) be arrested, and in the time periods/fictional narratives that Dungeons & Dragons typically mirrors, he could expect to lose a hand, be forced into literally back-breaking manual labor for more years than he’s likely to survive, or just get hung. His party would be expected to either aid in his capture or be treated as in league with him and treated to the same punishments.
Even if he doesn’t get caught, almost all of his party should not appreciate his antics, in-character. He’s behaving in ways that are decidedly Chaotic and non-Good, which should annoy the Lawful and Good sides of the spectrum, and for everyone else he’s just likely to get them all in trouble, get them involved in some stupid investigation or arrest or whatever when they have important things to be doing (the party does have important things to be doing, right?).
In other words, there is no real good reason why his party wouldn’t kick this guy out. He only causes problems.
And you should explain all of this to the players, out of character, so they can discuss the issue. Maybe they want a silly, light-hearted game where these kinds of antics go unnoticed and they can all laugh at the ridiculous shenanigans he got up to. Maybe they don’t realize that he’s eliminating any possibility of that. Maybe they’ve all been thinking the same thing (except for the player in question), and been waiting for you to call him on it. Regardless, get input from every player on how serious a game they want, how they feel the player’s behavior is or is not derailing the game, and so on. Find a compromise or consensus, and go with that. And if everyone else disagrees with you, you have to decide whether you are going to just accept that, or find a new group. You can’t tell them they’re all wrong and make them play as you want to play.