It's up to the DM, but there are two examples of monsters with hypnosis abilities we can draw guidance from.
The Kelpie (From White Plume Mountain in Tales from the Yawning Portal) has an ability called Drowning Hypnosis:
The kelpie chooses one humanoid it can see within 150 feet of it. If the target can see the kelpie, the target must succeed on a DC 11 Wisdom saving throw or be magically charmed while the kelpie maintains concentration, up to 10 minutes (as if concentrating on a spell). [...]
The Morkoth (VGtM, pg. 178) has an ability called Hypnosis:
The morkoth projects a 30-foot cone of magical energy. Each creature in that area must make a DC 17 Wisdom saving throw. On a failed save, the creature is charmed by the morkoth for 1 minute. While charmed in this way, the target tries to get as close to the morkoth as possible, using its actions to Dash until it is within 5 feet of the morkoth. A charmed target can repeat the saving throw at the end of each of its turns and whenever it takes damage, ending the effect on itself on a success. If a creature’s saving throw is successful or the effect ends for it, the creature has advantage on saving throws against the morkoth’s Hypnosis for 24 hours.
The Ultroloth's (MM, pg. 314) ability Hypnotic Gaze reads:
The ultroloth’s eyes sparkle with opalescent light as it targets one creature it can see within 30 feet of it. If the target can see the ultroloth, the target must succeed on a DC 17 Wisdom saving throw against this magic or be charmed until the end of the ultroloth’s next turn.
The Demogorgon from Out of the Abyss has a hypnotic gaze ability:
Demogorgon turns his magical gaze toward one creature that he can see within 120 feet of him. That target must make a DC 23 Wisdom saving throw. [...] The target is charmed by Demogorgon until the start of Demogorgon’s next turn. Demogorgon chooses how the charmed target uses its actions, reactions, and movement.
A Krasis from Guildmaster's Guide to Ravnica can have the Hypnotic Display ability:
As an action, the krasis creates mesmerizing colors and shapes around itself. Each creature within 15 feet of it that can see it must succeed on a Wisdom saving throw (DC 12 for category 1, DC 14 for category 2, and DC 18 for category 3) or be stunned for 1 minute. The stunned creature can repeat the saving throw at the end of each of its turns, ending the effect on itself on a success.
The precedent seems to be Wisdom, but the DM can choose otherwise if they have good reason to. The Kelpie's ability description is quite long, I'd prefer not to reproduce it here in its entirety (staying abreast of guidance in this meta answer), but the Morkoth's ability is more brief and gives a good template for what hypnosis could be.
[This section was made less relevant due to an edit to the question] Finally, there is also the crowd control spell, hypnotic pattern:
You create a twisting pattern of colors that weaves through the air inside a 30-foot cube within range. The pattern appears for a moment and vanishes. Each creature in the area who sees the pattern must make a Wisdom saving throw. On a failed save, the creature becomes charmed for the duration. While charmed by this spell, the creature is incapacitated and has a speed of 0.