I am interested in exploring the Pathfinder Campaign Setting line of products.
What is the best order to buy them in?
The Pathfinder Campaign Setting line, used to be the Pathfinder Chronicles line, is the set of GM supplements about the world of Golarion specifically. I have all of them since I've been a Pathfinder subscriber from day 1.
You should definitely start with the Inner Sea World Guide (ISWG), or, in a pinch, the original 3.5e Pathfinder Chronicles Campaign Setting (PCCS) book. And maybe a cheap PDF of the smaller Gazetteer to give out to your players so they'll read up too.
Then, I would get Gods and Magic (3.5 but still the definitive gods book) and Inner Sea Magic as the most widely applicable of the more 'general' (not region specific) books. There's others of this 'generic inner sea crunch" line - artifacts, prestige classes - but I don't find those of high value. Of special note is the Inner Sea NPC Codex - not the best first source for Golarion lore but in actually running games I find this and its big and littler brothers in the Pathfinder lines invaluable.
After that, I'd pick the region or regions you plan to set stuff in, and the other kinds of book (planets? darklands? planes?) that will support your campaign. Don't be afraid of the 3.5 Chronicles line, they haven't re-done those books since and they hit some of the big basics.
I find the Cities/Lost Cities/Towns/etc of Golarion line to be really good but I run very travel intensive games; if you don't you may find yourself just reading them for other-region flavor but not actually using them.
Lastly I'd look at the "* Revisited" books. They give a specific take on demons, dragons, flumphs, whatever, and they're usually quite good, but in terms of really helping with Golarion they're bottom of the heap.
Very last are the map folios - they're a vanity product, no real need for them. The maps are in the products themselves and the only reason to get them is to have a big ol' map to put on the wall or for rich players to pore over while roving around an adventure path.
There's a lot of Golarion, so you should focus in on what's relevant. For example, I run a pirate campaign set in the Arcadian Ocean that's 4 years old now. I therefore primarily use Riddleport info from Second Darkness, the Varisian info from the first APs and the ISWG, Nidal info from ISWG and Nisroch from "Cities of Golarion," Isles of the Shackles, Ilizmagorti from "Cities of Golarion," the Sun Temple Colony from "Lost Cities of Golarion," the Mwangi info from Heart of the Jungle, the Linnorm King book, Gods & Magic, and the Inner Sea NPC Codex.
I don't think there is a generally applicable answer for this question, as it mainly depends on your own preferences.
All I can really point you to is the obvious: Read the Inner Sea World Guide.
Once you are through that, you can expand whatever sparked your interest, be it a specific area, country, faction, deity with the appropriate products.