For example, let's say some cultists created totally new language that only people inside the cult know. It doesn't use any words from any other existing language. Does this spell still let mages understand the language, or is it left untouched as it is not even a normal language?
Yes, it does.
Comprehend languages says:
For the duration, you understand the literal meaning of any spoken language that you hear.
If these cultists are speaking the language, comprehend languages allows a listener to understand the literal meaning of what they are saying. A language is a language.
Astute cultists would focus their efforts instead on encoding an existing language.
Notice comprhend languages conveys the literal meaning of the words. If I encode my language into an existing language, say, through word substitution, comprehend languages will convey nonsense, as it only conveys the literal meaning of the words, not the encoded meaning. For a canonical example of such a language, we have Thieve's Cant:
During your rogue training you learned thieves' cant, a secret mix of dialect, jargon and code that allows you to hide messages in seemingly normal conversation.
Regarding Thieve's Cant, Lino Frank Ciaralli explains in this answer:
So while you may understand the literal message, "The rooster flies when the blood moon rises," you will not understand that this means that, "The man with the red scar on his neck is your target for assassination by midnight tonight."
This is akin to telling somebody to read the 3rd letter of the first paragraph in a book, in which it spells out a secret message to them. Comprehend languages would let you understand the paragraph, but would give you no insight whatsoever to the hidden code inside it.