I am not quite sure which skill scent would fall under. Our group just made our own category for it such as Spot, Listen, etc. That is working fine for use but I was just curious as if there was a specific one it was already suppose to be under?
Use a Wisdom check.
Sometimes a character tries to do something to which no specific skill really applies. In these cases, you make an ability check. An ability check is a roll of 1d20 plus the appropriate ability modifier. Essentially, you’re making an untrained skill check.
Usually, being able to perceive and recognize a scent involves feats and racial or class features like Scent, rather than skills.
What follows is purely speculation and my idea on how to manage it, since the rules stay silent.
Sometimes the scent is so strong that you'd need not to have a nose to be unable to peceive it.
Whether a character feels it is DM fiat. Recognition of the specific smell is a different beast.
If it's some smelly reagent, I'd suggest using craft (alchemy). If it's related to nature, a knowledge (nature) check would be in order. While recognizing some smell as something specific is probably related to Intelligence (memory and knowledge), a wisdom roll could be more appropriate to recognize something as "the same smell of that other room".
By the way, creating yet another skill a character needs to spend points in is IMHO unfair to the players. Using an untrained wisdom check (with appropriately low DCs) might be a better idea.
A normal person without any special olfactory abilities would use a Wisdom check to determine whether the person detects the scent. A character with special olfactory abilities could have a feat or racial trait that lets them use skills to detect scent (Search or Spot depending on the situation).
Example: Wolverine the mutant has his heightened scent of smell represented by a feat that lets him use Search or Spot for scents as well as the normal uses. Sitting in a bar, he smells his enemy Sabertooth (successful Spot check). After a fight, Sabertooth gets away. Wolverine tries to go after his scent but it is lost due to a nearby fire (failed Search check).
If a scent is already detected, but needs to be identified, use the approach suggested by Zachiel. There is often appropriate skill for the job (determining the material of a wooden box might be Craft(Carpentry) for example).
That is, there is no skill involved for the scent ability. According to the Monster Manual, the only skill involved with the scent ability is the Survival skill, and that's used exclusively for tracking using the scent ability.
Scent ability is largely binary--either working or not working--unless the scent-possessing creature is tracking by scent.
What you're likely looking at is this line: "Creatures with the scent ability can identify familiar odors just as humans do familiar sights" (MM 314), and assuming there must be some sort of check involved. There isn't.
The scent ability does exactly what it says on the tin, no more and no less. To paraphrase, it permits odor identification, detection of approaching enemies, the sniffing out of hidden foes, and tracking by sense of smell. All of which, except the first, are spelled out in the rules.
If you're letting it substitute for Spot checks or Listen checks, you've entered into house rule territory. If you're wondering what checks to use to determine if the scent-using character can discern a familiar odor, that's house rule territory, too. In the first case, it can't, and, in the second, there's no need.
I've played a PC with the scent ability in Pathfinder from levels 1 through 12 over the course of a year. I kept notes to determine which odors my PC was familiar with, and I made specific efforts to play the scent ability as if it were just as accurate (and inaccurate) as sight. Part of experiencing a new culture for the character was going around smelling the sights, partly because of the weird synesthetic nature of being able to say that and partly because it would make PC familiar with those odors were he to encounter them again.
It depends on the odor in question. If it's a question of whether or not the individual will detect an odor - as an indication that the party is about to be ambushed, for example, spot or listen (your call) would be appropriate. Remember, the rules are an abstraction. If you need justification, remember that smell is tied directly to the brain like no other sense so responses to it can be reflexive - perhaps it leads to the active use of other senses.
If it's a matter of detecting a trap by the odor of one of its components? This is included in the saving throw for the trap - especially reflex saves. The odd odor alerts the character a moment before the explosion, etc.
If it's identifying something by odor, whatever skill you'd use to create the thing (or otherwise identify it) is fine, knowledge, professions, etc.
And when all else fails, if nothing seems appropriate, a raw Wisdom check with a low DC is probably just fine.
By RAW, there's no skill for this. It's easy to think of skills where identifying some scents might be important -Survival for animals, Profession [Chef] for foods, and so on- but none of these are really concerned with smell in general.
At some point back in the 3.0/3.5 days -I think it was in Savage Species- they started working out ideas of just what it means to have a score in a given ability. This mattered most in the case of having a Con score (with a Con-less creature having either no body or no metabolism) but they did a small writeup for every score. Wisdom was the one they linked to the senses: any creature that could perceive its environment in any manner has at least one point of Wisdom.
For that reason, I'd be tempted to rule smells as a generic Wisdom check, possibly with synergy and circumstance bonuses for characters that should be especially familiar with whatever it is they are smelling.
If you don't want to use an Ability Check, you can use the Survival skill. I think it fits well, plus I found this which somehow confirms the idea:
A creature with the Track feat and the scent ability can follow tracks by smell, making a Wisdom (or Survival) check to find or follow a track.
A creature with the scent special ability can track using the survival skill and while there is no specific skill mentioned I use a listen like check to detect hidden foes. I was pretty sure that it's mentioned in the D&D v3.5 rulebooks somewhere.
Or you could just use the survival skill, you can track by scent using that why not sniff out other things as well using it.