The rules on "Using Different Speeds" state (emphasis mine):
[...] Whenever you switch, subtract the distance you've already moved from the new speed. The result determines how much farther you can move. If the result is 0 or less, you can't use the new speed during the current move. [...]
However, I'm unsure what the rules mean when they say "the distance you've already moved" especially when difficult terrain is thrown ininto the mix. The very next section is on "Difficult Terrain" and it states:
[...] Every foot of movement in difficult terrain costs 1 extra foot. This rule is true even if multiple things in a space count as difficult terrain. [...]
So what happens if a creature with a Walking Speed of 30 and a Climbing Speed of 20 begins their turn by walking through 15 feet of difficult terrain to the nearest wall (which is not difficult terrain)? Their remaining Walking movement will be 0, but will their remaining Climbing movement be 5 or 0?