Your precise quote comes from Fiend Folio rather than Book of Vile Darkness. While Book of Vile Darkness does have the infernal graft machine, whose additions apply the same −6 penalty, that book refers to them as “demonic additions,” as in
When characters with demonic additions deal with nonevil NPCs, they take a −6 circumstance penalty on all Charisma-based checks, such as Diplomacy and Bluff.
(Book of Vile Darkness, pg. 120)
While obviously very similar to your quotation, the exact quotation you offer, referring to “fiendish graft” instead of “demonic addition,” is in Fiend Folio on page 210.
Everything in this answer applies equally well to the Book of Vile Darkness (both are “3.25e” books as discussed below) and the demonic additions of the infernal graft machine (the slight differences in wording of the penalty don’t change anything), but I’m going to refer to Fiend Folio and fiendish grafts, instead, since Fiend Folio is the more substantial source on this subject.
Some things to keep in mind here:
Fiend Folio is considerably older than Tome of Magic, so there was no possibility that they considered binding checks when they wrote this rule.
Fiend Folio also predates 3.5e itself, even though it is nominally a 3.5e supplement. Such supplements are often referred to as “3.25e” and frequently have issues where they do not fully reflect the final rules of 3.5e. I have no evidence that is the case here, but something to consider.
The penalty doesn't apply to all Charisma checks relating to nonevil NPCs, but to “interacting” with them. What does interacting mean in this context? “Diplomacy, Bluff, and so on.”
What we basically have here is that there wasn’t any defined game term that referred to exactly what they wanted, and didn’t include anything else. So they tried to describe the kind of thing they had in mind, hoping that it would work as an “I know it when I see it” situation, but it never really did. For instance, if someone bedecked in fiendish grafts cast planar binding on a celestial, should that person take a penalty on the Charisma check to compel the celestial to do something? I can very easily see that going either way, and I’m sure the penalty applies at some tables and not at others.
The binding check made when forging a pact with a vestige might be similar to planar binding, so that question has relevance here. But the are also some significant additional concerns: does a vestige count as “nonevil”? And does it count as an NPC?
These are questions because vestiges do not actually exist. They are not a part of (this) reality, and they rely on binders to share in their existence. Thus, while Naberius certainly isn't evil, it isn't good, lawful, chaotic, or even neutral, either—and in context, “nonevil” here reads to me as “of an alignment that isn’t evil,” while vestiges are of no alignment—not even neutral. Likewise, while they are certainly controlled by the DM, and not by players, it is difficult to describe them as “characters,” per se. They don’t exist.
The long and the short of it is, the rules here are unclear. But the lore is crystal clear: no vestige is going to care even slightly what fiendish grafts you have or how they may or may not have “twisted” your personality. The penalty definitely should not apply.