Yes, hypnotic stare is an attack
Per page 208 of the Pathfinder Roleplaying Game Core Rulebook:
All offensive combat actions, even those that don't damage opponents, are considered attacks.
Thus, hypnotic stare would break the invisibility granted by the vanish spell.
On the RAW
Since you asked in particular for the RAW, I wanted to expand this answer to note some potential weirdness about it, and also to recommend that you and your group do not strictly adhere to the RAW, but consider some common-sense houserules regarding it. Here is the full passage regarding the definition of "attack."
All offensive combat actions, even those that don’t damage opponents, are considered attacks. Attempts to channel energy count as attacks if it would harm any creatures in the area. All spells that opponents resist with saving throws, that deal damage, or that otherwise harm or hamper subjects are attacks. Spells that summon monsters or other allies are not attacks because the spells themselves don’t harm anyone.
If adhering to the rules as written when implementing this passage in your game, you'll end up with some potential weirdness. While hypnotic stare likely wouldn't be affected, some things would be. For example, the magic missile spell could add a rogue's sneak attack damage to each of its missiles, or to a fireball, as magic missile is an attack, and sneak attack does not specify that it requires weapon attacks or attack rolls. It's a bit of an odd cascade of weird interactions.
The invisibility spell in D&D 3.5 used text like this in the spell itself to cover the scenario of non-attack-roll "attacks," but Pathfinder moved that passage to the general rules. I do not believe the game supports the use of this passage outside of interactions with the invisibility spell and similar effects, so I would discuss with your group how it should be applied.