For example Lets use the Marrutact from Sandstorm, it can cast as a 5th level wizard. if it then takes a prestige class that requires the ability to cast 1st level Arcane spells and advances arcane casting such as Abjurant Champion. assuming it meets all other prerequisites does the Marrutacts spellcasting improve when it takes a level in Abjurant Champion or does it gain no benefit?
You usually need to take 1 level in the base class first
Since prestige classes (almost always) require you to choose a class (that you already have levels in), and in most cases your innate spellcasting explicitly does not count as having any levels in the class you cast spells as, you cannot expect a prestige class to help here.
Thus, when you take a level of wizard as a marrutact (or a level of sorcerer as a dragonspawn or a level of bard as a gloura), this stacks with (improves) your innate spellcasting, and means you have membership in that class for the purposes of choosing it with a prestige class. Since both the innate spellcasting and the prestige class levels stack with the base class, your total “effective” spellcasting level includes all three sources.
Note that for creatures like angels that cast as clerics and have domains, real-class-membership has further benefit:
Many thanks to @Forrestfire and @Taveena for helping me track down these rules.
But there really isn’t a great reason to require the 1-level dip in the base class
Ultimately, I’ve never met a DM who actually required that one level, and refused to allow the prestige class to just work. It seems appropriate and I’m dubious that there are any cases where it is broken to do so (at least, assuming the base creature wasn’t already broken on its own; there are definitely cases of that – Black Ethergaunt, I am looking at you).
Select prestige classes "add caster levels to an existing class"
Page 315 of Monster Manual 1 states:
So unless the monster specifically declares that it is a member of a class (not just "casts spells as class x"), there is no existing class to add caster levels to.