Yes they could still use Innate Spellcasting while using Shapechanger; a GM may decide that the creature can no longer provide certain spell components and thus would not be able to cast certain spells
The Shapechanger feature states:
[...] Its statistics are the same in each form...
We know the following from the question "What all is included in a creature's game statistics?":
What counts as statistics for monsters is defined in the Monster Manual introduction in the section "Statistics". It begins by saying (page 6):
A monster's statistics, sometimes referred to as its stat block, provide the essential information that you need to run the monster.
We have here a general rule that is easy to apply: if it is in a stat block then it is considered part of that creature's statistics.
The book describes each aspect of the statistics in a short sub-section (pages 6-11).
- Armor Class
- Hit Points
- Ability Scores
- Saving Throws
- Vulnerabilities, Resistances, and Immunities
- Special Traits
- Limited Usage
We also know from the Monster Manual (page 10) that Innate Spellcasting counts as a Special Trait (It is the first trait listed in the list of Special Traits).
Thus when you use Shapechanger you maintain your statistics which would include your Special Traits which would then include Innate Spellcasting.
That said, as the question "Does innate spellcasting by creatures have verbal or somatic components?" explains, even Innate Spellcasters still need to provide components. This would still apply while using the Shapechanger feature, so their new form can only cast spells for which it can provide the spellcasting components.
Note that this creature's statblock differs from other creatures such as the Yuan-ti Mind Whisperer, whose statblock includes:
Innate Spellcasting (Yuan-ti Form Only).
There is no similar restriction on the Innate Spellcasting feature of the Anchorite of Talos.