To infuse an item, you need the nonmagical version of that item.
The artificer's Infuse Item feature states (TCoE p. 12, E:RftLW p. 57, WGtE p. 179; emphasis mine):
At 2nd level, you gain the ability to imbue mundane items with
certain magical infusions. [...] Whenever you finish a long rest, you
can touch a non-magical object and imbue it with one of your
artificer infusions, turning it into a magic item. An infusion works
on only certain kinds of objects, as specified in the infusion’s
description.
The Artificer Infusions section at the end of the class description states (TCoE p. 20, E:RftLW p. 61, WGtE p. 181; emphasis mine):
The description of each of the following infusions details the type
of item that can receive it, along with whether the resulting magic
item requires attunement.
And finally, the description of the Replicate Magic Item infusion states (TCoE p. 22, E:RftLW p. 63, WGtE p. 182; emphasis mine):
[...] Alternatively, you can choose the magic item from among the
common magic items in the game, not including potions or scrolls.
In the tables, an item’s entry tells you whether the item requires
attunement. See the item’s description in the Dungeon Master’s Guide
for more information about it, including the type of object required
for its making.
Each of these portions I've bolded make it clear that you need a nonmagical version of the item to infuse, whether you're using Replicate Magic Item or one of the other artificer infusions.
Even though the Dungeon Master's Guide doesn't actually say "[X] is the type of item needed to make [Y]" for any listed magic item (as far as I know), this sentence seems to refer to the fact that each magic item that can be replicated is essentially an enhanced version of a clearly identifiable mundane item.
For instance, boots of elvenkind are boots, so the clear implication is that you would need mundane boots to infuse if you want to use Replicable Magic Item to make boots of elvenkind. This follows the same logic as the Boots of the Winding Path infusion (TCoE p. 21, E:RftLW p. 62, WGtE p. 181), which does explicitly state:
Item: A pair of boots [...]
In the case of your example, the description of cast-off armor states (XGtE, p. 136; emphasis mine):
Armor (light, medium, or heavy), common
Thus, you need a nonmagical version of whatever kind of armor you want to turn into cast-off armor. If you want cast-off half plate armor, you need a nonmagical set of half plate armor to imbue with that infusion.
The Replicate Magic Item infusion doesn't create a magic item from thin air - it gives you the ability to imbue a certain mundane version of that item with the necessary magic to function as that magic item.