This requires interpretation by your DM
The text you quote is the entirety of rules text for the item, so if it is ambiguous how to handle it, your DM will have to make a ruling on it. To cite Xanathar's Guide to Everything, page 5:
Here's the path the game takes: it lays a foundation of rules that a DM can build on, and it embraces the DM's role as the bridge between the things the rules address and the things they don't.
To your first question, what "you remain there" means -- I think this part is pretty clear. The robe transports you to the astral plane in general, not a specific location of the astral plane, so "there" just refers to the astral plane, not a specific location of the astral plane, and this only means you remain on the astral plane.
The second question is if you can leave by other means, or if "you remain there" means you are trapped there and can only leave by using an action with the robe, or if the sentence just aims to express that you can use an action to return. This is less clear. The robe transports you to the astral plane, and can transport you back, it does not say that you cannot leave, only that you remain (that is are not returned) "until you use an action to return"; the issue for this reading is that the "You remain there until" part of the clause would not be required, the item could just state "You can use an action to return". As you could argue either way, this is not clear, and your DM must rule on it. I would rule that you can return by other means, too.
Then lastly, if you allow the caster to leave, what happens if they use the second action? I think if they left by some other means, they did not remain there, and as they only can use that action when they "remain there until you use an action", they cannot use that action. If they return to the astral by some other means, it's even less clear if they then could use it. They have not really remained there. I'd probably rule that you can only return if you had not left, but this is certainly also so ambiguous that you need to ask your DM.