It behaves like any other creature would.
The PB states that any creature that moves out of your reach provokes an opportunity attack (195):
You can make an opportunity attack when a hostile creature that you can see moves out of your reach.
If the specter moves through a creature and out of their reach, then it will provoke an opportunity attack. If it passes through a creature but stays within its reach, it does not provoke an attack. This is, of course, the same for any other creature.
If a creature can avoid opportunity attacks, it's explicitly listed in the listing. For example, the Giant Owl (MM 327):
Flyby. The owl doesn't provoke opportunity attacks when it flies out of an enemy's reach.
Specters do not have such an ability, so they are subject to opportunity attacks if they move out of a creature's reach.
Compare to a ghost
It's worth noting that a ghost has a similar incorporeal movement ability and an Etherealness ability, which truly allows it to float through anything freely (MM 147):
Etherealness. The ghost enters the Ethereal Plane from the Material Plane, or vice versa. It is visible on the Material Plane while it is in the Border Ethereal, and vice versa, yet it can't affect or be affected by anything on the other plane.
This ability would allow the ghost to ignore any opportunity attacks (though it would still technically trigger them).