I have faced similar situation several times in our 4 years long GURPS fantasy campaign.
Half year after start, a guy playing a spy joined. His goal was to find his lost brother, and he quickly found the brother was kidnapped by common enemy. The guy left the party later, but he fit in well, especially when he could use his sneaky abilities.
More than two years after start a girl joined us with a character of priestess of a death God. We made a detailed backstory, a common goal and voice of her god suggesting to join the party. Since then, I made sure to include some demons, undead or black magic (the character is one of the best exorcists in area) in every session she plays. She had no links to PCs (except for common goal with one of them) or important NPCs, but desire to rebuild her god's cult and getting powerful allies in first few sessions helped her to integrate to the party quickly.
Few months after her, a guy joined as a scout. He left after two sessions - I think he expected something more similar to dungeoncrawling than to our campaign featuring espionage and intrigue (and war too, but he joined during a winter truce in the campaign).
A year after the priestess, a guy joined as a noble, a brother of one of the party's most important allies, who just returned from a long-term diplomatic mission. He quickly became one of the leading characters in the party.
Half year after that (almost three years after start) a mage joined the group. The character's biggest goals are training and research, but adventuring with the party is the right challenge for both the player and the character. Getting accustommed with the world and and rules wasn't easy for him, but he is one of the most active players from the very start.
Half a year ago, a new guy playing scout/ranger joined. The character simply inherited the noble's agenda (he was linked to a PC of one of the new players), but he has no goals of his own. He is not a very active player and I'm afraid he wouldn't stay for long.
Now I can induce few rules (they are somewhat similar to what mwyzplk suggested):
Motivation is crucial. The character needs agenda, which is interresting for the player (optimally the player's idea) and is either common or easily joinable with the agenda of the party or at least one other PC. If the player don't recognize his character's agenda as his own or if it's not weaved into the tapestry of party's goals (after more than a year, party's goals must be complex), the player would likely leave the party, or at least they wouldn't be interrested in the game, like your new player. I guess this is where you made the mistake. Being different (as DampeS8N suggested) could help, but still it's a passive goal, so it's not as easy to motivate the player as an active character-driven goal.
Spotlight is number two. Give the character scenes where they are the best expert in the party - it motivates the player and earns the character respect of the old party members. The worst situation is when there's already a PC of the same class and higher level, but these spotlight abilities don't have to be written in character sheet - you devil from hell can have information that others can't easily learn without him. Artifact mentioned in lisardggy's answer is another good example.
Also, help the player overcome the newcomer shock. There are many details in the game world to learn and many internal jokes in the party. Tell the player as much as possible before they join and keep explaining during game. Crucial point here is whether old players are helpful, whether they try to explain party's internal jokes and other difficult things to the newbie - GM can't make all the work by themself. Our group is excellent in this aspect, but still one player left because he didn't really want to adopt party's tactics and other customs. If your old players don't try to invite the newbie and the new player is not extroverted enough, this might be the core of the problem.
What to do in your situation? Tell your players to help the newcomer and sit with the new player to find some motive he would like. Ask him, what he expects, what he would like to do. If it's not reconcilable with his character, then replace the character, but this is unlikely. Then think how to give him what he wants, and ideally to join his goals with party's.
If the player just wants to honor his character's backstory (i.e. he is a "myguyist" of some sort), just state that the archvillain is an important ally (or even a master) of a rival faction in Hells, and let the party discover a plot arranged by another hellish faction. This will teach him to pay attention to mere mortals! Or bring some old friend from Hells, who would just visit him and ask what he found about this plane of existence - tell him, that ignorance is not good, even for a Hellboy like him.
Also, NPCs should be interrested in such a character, and the interrest doesn't have to be in form of assassination or exorcism attempts. Some should be interrested in him in the positive way - another encouragement to start involving in relationships with NPCs.