I was wondering if you can plane shift to a plane you never visited since you seem to need to specify a destination?
You don't need to visit a plane to go there.
The only thing that the description of plane shift says about where you arrive is:
you appear 5 to 500 miles (5d%) from your intended destination
Compare this with teleport, which says:
You must have some clear idea of the location and layout of the destination.
Teleport also has a bunch of other information about targeting a location, which says (among other things) that you need to at least see a place to teleport there. Since some transport magic specifically says that you need to see a place to go there, it implies that spells that don't say you need to see a place to go there, don't have that requirement.
The fact that plane shift says that you arrive near your destination, you do need to have a destination. However, this can be defined any way that your DM considers precise enough to count as a 'location'. For example, you could say that you wanted to go to a particular githyanki outpost in the Astral Plane, or a particular efreet keep on the Plane of Fire that you've heard of. As long as you can name a place, you can plane shift there.