Generally, yes, doing anything but the one task at hand - like moving, dodging blows and looking for openings - would constitute as "distracted" in regards to listen (and some other skill) checks. But listen is a special case, when it comes to invisible things: it has its own rules.
Let's explain that stuff a little: Combat is quite noisy in itself, as whenever a physical attack hits something, more noise is created than a combattant that just moves creates. That is, the noise is made when the attacks connect to the target or their parrying weapon, so the base DC to detect the general position of an invisible, fighting creature is 0 (and 20 to pinpoint). But if they are not engaging a specific PC (which, as engaged, automatically knows where it comes from as long as reach isn't involved), the distance to the invisible creature makes it often harder to directly pinpoint the combattant (+1 per 2 squares). So far the part on the invisibility status rule block.
But then there is the +5 for distraction on listen itself: I would not apply those, as a) the +1 per 10 feet/2 squares is specifically noted on the invisibility table and b) the +5 for being distracted is not. My interpretation of this is, that it is a specific case, that only for pinpointing the invisible creature in combat this penalty does not apply, even if it is realistically bogous and for all purposes should apply, if only because you make so much noise yourself. But the rules don't say so.
HOWEVER, if the invisible creature is not (yet) part of the combat, then I would just add it up: Move Silently + 5 + 1d20 (-movement mod) + distance.