Torso and body are separate slots, and you can wear torso items along with armor, e.g. a vest of resistance under +1 chain shirt or whatever. However, robes are defined as body-slot items in the Dungeon Master’s Guide, not torso-slot. In that book—which is the primary source on magic items—individual items don’t even list a slot per se. The robe of eyes on page 265, for example, doesn’t mention any slot at all. Instead, it’s a body-slot item just because it is a robe.
In later books, they started including slots in each item, just to be clear. Not a lot of people know what a “chasuble” is, for example, and even if one does, it’s not immediately clear what slot it should use (my first guess would have been body, but it’s actually throat, oddly enough). But the problem here is that we know what slot a “robe” is—the Dungeon Master’s Guide explicitly tells us that’s a body-slot item.
So now we have Dungeon Master’s Guide saying all robes are body-slot items, and Complete Champion describing a robe of the vagabond as a torso-slot item. The game does have rules for adjudicating discrepancies like this, saying you should go with the rule in the “primary source.” Which is (explicitly) the Dungeon Master’s Guide for magic items in general—if Complete Champion said robes in general were torso-slot items, we’d ignore it, because Dungeon Master’s Guide is primary for that. But Complete Champion is the primary source—the only source—for the robe of the vagabond. That is a conundrum.
It basically comes down to this: do you believe that Complete Champion was explicitly saying that this robe was special, and used the torso slot instead of the usual body slot? If so (and if the DM agrees), then yes, you can wear it with armor. If not, however, and you think this was just a mistake, you should ignore it and treat it as a body-slot item—and then you can’t wear it with armor. Considering that its text description makes no mention of this special property, I for one think it is most likely that this was just a mistake.