In 5e, familiars are just as dumb as the base animal they emulate. Truly you can communicate and issue orders, but at most they behave like a very well trained animal.
Is there a way to permanently increase their INT?
The Headband of Intellect is a magical item that raises the intelligence score of its wearer to 19. While this only lasts until it is removed, it should be permanent enough for your purposes. It does not grant the wearer the ability to speak or read, however.
Using the downtime training rules, your 19-Int familiar could learn a language, though ask your DM how well an animal can actually form words with its anatomy. At the very least, it would be able to speak well with you telepathically.
If you don't think that this is permanent enough, these could be followed by reading a Tome of Clear Thought (since it now knows how to read). Multiple Tomes might be required to raise its base intelligence up to the desired level. However, this does raise a few questions such as what happens if the familiar takes on a new form after reading the Tome? I'd recommend avoiding this whole mess by sticking to the Headband and language learning.
They're smart enough to follow orders, and not much else. Even if there was a way, I see no reason why you'd want to. Most familiars don't understand languages, and if you want to check out illusion magic or something else with them, you can perceive through their senses, rather than them telling you about it. If you want a smarter familiar, the warlock-exclusive ones have INT scores around 8 or so.