Yes, if spirit = soul
Let's look at what this specter is:
Starting at 6th level, you can curse the soul of a person you slay, temporarily binding it to your service. When you slay a humanoid, you can cause its spirit to rise from its corpse as a specter... (XGtE)
The soul/spirit is changed temporarily; it remains the soul of the slain character.
Is the creature's spirit the same thing as the creature's soul?
Humanoids have souls, and presumably some undead have souls (what does a lich keep in its phylactery?). In this case a spirit is being temporarily transformed into a creature.
Does the creation of a specter now establish another soul?
If it did that raises problems for this working, however, the creation of another soul looks to be a level of power beyond a 6th level character ability (and is overly complicated). Beyond that, the text of this ability suggests that the soul and the spirit are the same, given that the spirit goes to the afterlife, which is where souls are before the raise dead spell can bring a willing soul back.
The specter remains in your service until the end of your next long rest, at which point it vanishes to the afterlife. (XGtE, p. 56)
Does anything in revivify prevent this?
Revivify as a spell has very few restrictions.
You touch a creature that has died within the last minute. That creature returns to life with 1 hit point. This spell can’t return to life a creature that has died of old age, nor can it restore any missing body part. (PHB)
As long as revivify is cast within the one minute of the party member dying, by touching the dead PC (the soul's standing right there), then if soul = spirit the body and soul are reunited.
Caution; make sure to touch the dead PC; the specter isn't dead -- it's undead
A specter is available under this power because a dead creature's soul/spirit was available. The PC is not at 0 HP; the PC is dead. Based on the raise dead spell (and other spells) there is a separation of body and soul at death).
If the creature’s soul is both willing and at liberty to rejoin the body, the creature returns to life with 1 hit point. (PHB, Raise Dead; see also body and soul separation in Reincarnate, Resurrection text).
What happens to the specter?
Since it was the soul of the dead PC that was acting as a specter, the specter vanishes with the casting of the revivify spell as the party member is brought back to life. It is very clear that the soul and body are connected in living creatures based on spells that discuss the existence of the soul: Astral Spell, Clone, Magic Jar, Raise Dead, Resurrection, Reincarnate, et al.
Does being a specter prevent this from working?
No, the specter should be able to go through walls. @Axoren noted that in the rules for objects (DMG), Castle Walls are cited as being objects (specifically ones with a damage threshold). A wall is thus an object for purposes of a specter going through objects:
Incorporeal Movement. The specter can move through other creatures and objects as if they were difficult terrain. It takes 5 (1d10) force damage if it ends its turn inside an object. (MM, Specter)
Beyond that, the door to be opened would be an object that the specter could pass through. (Thanks to @Pharap for that point)
Also, see the RP opportunity below for side effects on being a specter for a little while.
Specter / Medium undead, chaotic evil (etc)
Nothing in the monster description indicates an adverse interaction with revivify.
Creature type? No restriction.
Undead are once living creatures brought to a horrifying state of undeath through the practice of necromantic magic or some unholy curse. Undead include walking corpses, such as vampires and zombies, as well as bodiless spirits, such as ghosts and specters.
Since the condition is temporary, you could view this transformation into a specter as a limited form of polymorph.
Humanoids are the main peoples of a fantasy gaming world (MM, Humanoids)
... and humanoids have souls, which is what makes this whole exercise possible if spirit = soul.
If the spirit = the soul, then the self sacrificing character's soul's alignment was changed for a little while to chaotic evil (per the specter spell description). This opens up some interesting side effects (work with your DM on this) if the character's original alignment wasn't chaotic evil.
How serious is death in your campaign?
If you tried this in Tomb of Annihilation, your the PCs might be in for a jolt. Likewise in Curse of Strahd. Some tables, and some DMs, treat death (not dropping to 0 HP) as a significant traumatic and violent experience. Repeated abuse of this (clever) combination could lead to serious side effects: for example, insanity. You can review the tables in the DMG on insanity, or adapt the insanity element from the Out of the Abyss published adventure for models of how insanity can influence game play.