It's debatable whether they would be affected at all
There are numerous spells that do damage or otherwise inconvenience a creature when that creature moves, or is moved, into their area of effect. Since you used Wall of Fire as an example, let's examine the text that spell uses:
Wall of Fire
A creature takes the same damage when it enters the wall for the first time on a turn or ends its turn there. (PHB, p. 285)
This movement does not need to be voluntary: if you are shoved into a Wall of Fire, or walk into it yourself, you still "enter" the wall. However, note the similar (but not identical) text of Prismatic Wall (PHB, p. 269, bold added):
When a creature attempts to reach into or pass through the wall, it does so one layer at a time through all the wall’s layers. As it passes or reaches through each layer, the creature must make a Dexterity saving throw or be affected by that layer’s properties as described below.
This "attempts" is crucial. If a creature is grappled and moved against its will, it did not "attempt" to move in the direction it is moving: it had no choice in the matter. For a creature to "attempt" to pass through the wall, it must be moving through it (or reaching into it) voluntarily.
This is also complicated by the fact that the second sentence in the quote above doesn't require intent, but simply that a creature "passes through" or "reaches through" a layer of the wall. It is a DM's prerogative to decide which of these sentences takes precedence in this situation.
What happens if you're pushed/grappled into the wall?
It's very hard to say. Strangely, none of the layers of the wall prevents a creature from physically passing through it while they are active (layers prevent things like magical and nonmagical ranged attacks from penetrating the wall). Nor is there any text that directly states that the wall is solid. It is described as "opaque", but that simply means you cannot see through it.
It would be a DM's call what would happen if you shoved a creature towards the wall, or attempted to run through the wall with a creature grappled. A case could be made for the idea that the wall is solid when you are thrown into it, because of the "attempts" mentioned mentioned earlier: this time, "attempts" is significant not because it indicates willing movement, but because it suggests a possibility of failure. If you "attempt" to move through something, it does somewhat imply that moving through it is not guaranteed, which might imply some level of solidity. Again though, that's very much a DM's call.
One thing seems clear to me: it does not seem likely that the wall is meant to be both harmless and intangible to creatures who are pushed through it involuntarily. If that were the case, a group of enemies could simply shove each other through it and avoid its effects altogether (except the unlucky one who is last).
What about the Blindness Condition?
(Credit to HellSaint for mentioning this in a comment). Passing into or through the wall is not the only way the wall could affect a creature (PHB, p. 267-269 [sic]).
...If another creature that can see the wall moves to within 20 feet
of it or starts its turn there, the creature must succeed on a
Constitution saving throw or become blinded for 1 minute.
This effect is one place where the rules seem somewhat clearer. The text says "moves" rather than "is moved", but forced movement is still movement. Likewise, the text of Wall of Fire requiring a character "enter" the area is similar grammatically to the text here requiring the creature to "move." And crucially, there is no text in this section restricting this particular affect to happening only the first time that a creature moves into its area on a particular turn.
Although there is no designer answer to this question as yet, the Rules As Written (RAW) seem to indicate that you could drag a creature close to the wall (within 20 feet) and subject them to the saving throw to avoid blindness. And conceivably, if the creature succeeds, you could then drag them away from the wall, and towards it again all on the same turn, subjecting them to the same saving throw once again.
Of course, you'd need to spend a 9th level spell slot to do so.