They can move around freely.
It's binbeen my experience that once something is invisible in the heat of battle that's all there is to it. They can move around freely. However, if when they pass close to one of your party members you want to roll against their passive perception to see if they hear them move past that would be interesting. I
I may use that myself now.