The target makes a Wisdom saving throw.
What the "spellcasting ability" is:
Any character that casts spells has a "spellcasting ability". For spellcasting characters this is determined by their class. E.g. Wizards use Intelligence. Clerics use Wisdom. And so on. (Sometimes a character may also be able to cast some spells due to their race or other feature - in this case, the description will say "Your spellcasting ability for this is [ability]"; which only applies to those particular spells).
The spellcasting ability is, among other things, used to determine the difficulty (DC) of resisting those spells. This is generally 8 + proficiency bonus + spellcasting ability modifier
The same rules theoretically apply to spellcasting monsters. In their description it will tell you what their spellcasting ability modifier is just in case it becomes relevant, though it will always tell you the save DC anyway.
For mindflayers this is Intelligence. In the mindflayer's description it tells you that the DC is 15. (This is equal to 8 + their Intelligence modifier of +4 and their proficiency bonus of +3)
How you determine what saving throw to use:
Put simply: It will tell you in the spell description.
Example: Detect Thoughts requires a Wisdom saving throw
An unfortunate victim of a Mindflayer casting Detect Thoughts must make a Wisdom saving throw against a DC of 15.