Setting aside the obvious common-sense explanation to examine the RAW, teleportation effects move the subject from one location to another, without passing through any intervening space. Per the teleportation rules:
Teleportation is instantaneous travel through the Astral Plane. Anything that blocks astral travel also blocks teleportation.
As mentioned in the quote, it's possible for some effects to obstruct your movement to or from the Astral Plane (such as a dimensional lock spell), but those have their own rules, separate from the mechanics for difficult terrain. Difficult terrain effects could be present on the Astral Plane itself, but wouldn't have any meaningful effect on your teleportation - the Astral Plane is timeless, so no matter how much you might get "slowed down" while you're there, you'd still return to the material at the exact instant you left, as per the rules for teleportation and the descriptions of the various teleportation spells.
Somewhat related to teleportation is ethereal movement, offered by spells such as ethereal jaunt. These effects are non-instantaneous and use the normal movement rules, so you could still be slowed down by difficult terrain, though that would be rather unusual (ethereal movement can generally bypass the material plane's difficult terrain and even walls - instead, you'd have to run into difficult terrain that exists on the ethereal plane itself, such as an ethereal creature using Vigilant Keeper's Stance).