For example Lets use the Marrutact from Sandstorm, it can cast as a 5th level wizard. if it then takes a prestige class that requires the ability to cast 1st level Arcane spells and advances arcane casting such as Abjurant Champion. assuming it meets all other prerequisites does the Marrutacts spellcasting improve when it takes a level in Abjurant Champion or does it gain no benefit?
You usually need to take 1 level in the base class first
Sometimes a creature can cast arcane or divine spells just as a member of a spellcasting class can. [...] A spellcasting creature is not actually a member of a class unless its entry says so
Since prestige classes (almost always) require you to choose a class (that you already have levels in), and in most cases your innate spellcasting explicitly does not count as having any levels in the class you cast spells as, you cannot expect a prestige class to help here.
A spellcasting class is an associated class for a creature that already has the ability to cast spells as a character of the class in question, since the monster’s levels in the spellcasting class stack with its innate spellcasting ability.
Thus, when you take a level of wizard as a marrutact (or a level of sorcerer as a dragonspawn or a level of bard as a gloura), this stacks with (improves) your innate spellcasting, and means you have membership in that class for the purposes of choosing it with a prestige class. Since both the innate spellcasting and the prestige class levels stack with the base class, your total “effective” spellcasting level includes all three sources.
Note that for creatures like angels that cast as clerics and have domains, real-class-membership has further benefit:
A creature with access to cleric spells must prepare them in the normal manner and receives domain spells if noted, but it does not receive domain granted powers unless it has at least one level in the cleric class.
Many thanks to @Forrestfire and @Taveena for helping me track down these rules.
But there really isn’t a great reason to require the 1-level dip in the base class
Ultimately, I’ve never met a DM who actually required that one level, and refused to allow the prestige class to just work. It seems appropriate and I’m dubious that there are any cases where it is broken to do so (at least, assuming the base creature wasn’t already broken on its own; there are definitely cases of that – Black Ethergaunt, I am looking at you).
Select prestige classes "add caster levels to an existing class"
Page 315 of Monster Manual 1 states:
"Sometimes a creature can cast arcane or divine spells just as a member of a spellcasting class can...A spellcasting creature is not actually a member of a class unless its entry says so."
So unless the monster specifically declares that it is a member of a class (not just "casts spells as class x"), there is no existing class to add caster levels to.