Shard Swarm Doesn't Make Sense to Begin With
If you're treating Shard Swarm as a teleport, which is what it says in the effect, but roleplaying it as the shardmind disassembling and reassembling, then you're already doing something that doesn't make physical sense.
Teleport requires line of sight, but not line of effect. In other words, you can teleport through a window. How does that makes sense if the pieces of your shardmind are physically moving from one location to another?
Neil Slater pointed out another problem, which is how does the shardmind's equipment move? Do the little crystal pieces pick up the equipment and carry it? Could a creature grab your sword out of the air as some little crystals try and carry it?
If you're fine with all this, then sure, the arm goes with him.
Maybe Roleplay it a Little Differently
If these inconsistencies bother you, then I would roleplay the teleport as a teleport. Maybe the shardmind loses some cohesion beforehand, but they then disappear and reappear elsewhere. Then all those problems disappear, and the arm just teleports with the rest of the character.
Not an Isolated Issue
These kind of issues seem to be a recurring problem with many of the 4E powers, which make more sense as moves in a game than they do as actions a character would take in a verisimilitudinous world. The Eladrin's Fey Step can exhibit similar inconsistencies, especially in Eberron where the Eladrin are supposedly "searching desperately for a way back to the Feywild". Just use Fey Step guys, duh.
I would recommend making a conscious decision to either ignore these issues and go with the rules and flavor as written, or to massage things so that your character makes sense to you, keeping in mind how your play group in general feels about it.