The MM says
The mind flayer's innate spellcasting ability is Intelligence (spell save DC 15). It can innately cast the following spells, requiring no components: At will: detect thoughts, levitate l/day each: dominate monster, plane shift (self only)
Detect Thoughts and Dominate Monster call for Wisdom saving throws, but because their spellcasting ability is Intelligence, does the target make an Int or Wis saving throw?
What the "spellcasting ability" is:
Any character that casts spells has a "spellcasting ability". For spellcasting characters this is determined by their class. E.g. Wizards use Intelligence. Clerics use Wisdom. And so on. (Sometimes a character may also be able to cast some spells due to their race or other feature - in this case, the description will say "Your spellcasting ability for this is [ability]"; which only applies to those particular spells).
The spellcasting ability is, among other things, used to determine the difficulty (DC) of resisting those spells. This is generally 8 + proficiency bonus + spellcasting ability modifier
The same rules theoretically apply to spellcasting monsters. In their description it will tell you what their spellcasting ability modifier is just in case it becomes relevant, though it will always tell you the save DC anyway.
For mindflayers this is Intelligence. In the mindflayer's description it tells you that the DC is 15. (This is equal to 8 + their Intelligence modifier of +4 and their proficiency bonus of +3)
How you determine what saving throw to use:
Put simply: It will tell you in the spell description.
Example: Detect Thoughts requires a Wisdom saving throw
Summary:
An unfortunate victim of a Mindflayer casting Detect Thoughts must make a Wisdom saving throw against a DC of 15.
The target makes whatever saving throw the spell asks for. Spells explicitly call out when the spellcasting ability is used (like Counterspell asking for a spellcasting ability check).
Detect Thoughts is actually normally unavaliable for classes using Wisdom as spellcasting ability. It is avaliable for Bards, Sorcerers (CHA) and Wizards (INT).
The "spellcasting ability" refers to which of the creature's statistics is used to determine its spellcasting - consider the fact that whether a Fireball is cast by a Wizard (Intelligence) or Sorcerer (Charisma), the saving throw remains Dexterity.
Instead, the monster's entry lists Intelligence as its casting stat so that you, the DM, know that should your players somehow make the monster less intelligent, the DC of its spells should go down.
