The rules on this matter are subject to some level of interpretation. Since ghosts and possessed monsters are usually both controlled by the DM it is perfectly reasonable to just make a ruling on what makes more sense for the group/setting (or simply what is more fun) rather than going with a RAW interpretation (except, perhaps, for the section on armor).
That said, let's check what the rules are on what the ghost does or does not get access to during a possession (all emphasis mine):
The ghost now controls the body but doesn’t deprive the target of awareness. The ghost can't be targeted by any attack, spell, or other effect, except ones that turn undead, and it retains its alignment, Intelligence, Wisdom, Charisma, and immunity to being charmed and frightened. It otherwise uses the possessed target's statistics, but doesn't gain access to the target's knowledge, class features, or proficiencies.
More relevant to this question are the monster statistics called "Special Traits", which in general the ghost has access to and include monster spellcasting. Both the section in the monster manual (p. 10) and the 5e SRD have detailed the following sub-sections on spellcasting:
Short answer: Yes, might depend on the DM (I would personally allow it) but see the section on armor below.
This is simply a special ability the monster is able to use that happens to involve the casting of a spell.
A monster with the innate ability to cast spells has the Innate Spellcasting special trait. (MM p.10)
An argument could certainly be made that the ghost might not "know" how to perform the specific components of a spell, and while a DM may rule so I would personally disagree. After all an "innate" ability suggests that it is naturally a part of the creature itself. The ghost similarly does not really "know" how to use any other ability or action that it does not already possess (see, for example, a werewolf's shapechange or the half-dragon's dragon's breath).
Short answer: No
This is a matter that might be more problematic since the sections in the monster manual and SRD are slightly different (all emphasis mine):
A monster with the Spellcasting class feature has a spellcaster level and spell slots, which it uses to cast its spells of 1st level and higher (as explained in the Player's Handbook) (MM p.10)
A monster with the Spellcasting special trait has a spellcaster level and spell slots, which it uses to cast its spells of 1st level and higher (as explained in the Player’s Handbook). (5e SRD p.258)
This means that spellcasting for monsters is a "class feature" according to the monster manual (no word on this was found in the MM errata), automatically excluding it from the list of traits that a ghost would get access to. However, according to the SRD it is simply a "special trait" which should be further considered. Regardless, both entries point to the Player's Hanbook for the rules on known or prepared spellcasting which state, among other things
Before a spellcaster can use a spell, he or she must have the spell firmly fixed in mind, or must have access to the spell in a magic item. (PHB p. 201, emphasis mine)
The ghost is only controlling the possessed creature's body (see quote above) and not its mind so even if you want to go with the SRD entry and call it a special trait, casting known or prepared spells would not be possible for the ghost.
Do note that going by the SRD interpretation the ghost still gets access to the creature's spell slots, which it can use for other purposes (for example, the Priest's Divine Eminence ability.)
Short answer: No
A monster that casts spells using only the power of its mind has the psionics tag added to its Spellcasting or Innate Spellcasting special trait. (MM p.10, emphasis mine)
Again, the ghost is only controlling the victim's body and not its mind so casting spells with the power of its mind would not make sense.
Regardless of the specific way of spellcasting, something that might be easy to overlook is the fact that a possessing ghost also gets the target's equipment (part of the statistics). With no default equipment of its own a ghost usually has no armor (or weapon) proficiences, though a DM might want to grant it. This causes a ghost possessing a creature wearing armor (such as the priest which has a chain shirt) to be subject to the rules on wearing armor without proficiency:
If you wear armor that you lack proficiency with, you have disadvantage on any ability check, saving throw, or attack roll that involves Strength or Dexterity, and you can't cast spells. (PHB p. 144, emphasis mine)
This stops the ghost from casting any kind of spell unless it gets rid of the armor (see the table on PHB p.246 for how long that would take).