When would something be evil, but not evil enough for a paladins smite evil to function?
Would it function off of detect evil chart as when something is evil enough?
In Pathfinder, all creatures have an alignment. This alignment can be evil, or it can be not evil. There is no "kind of evil"; there is no "evil but not evil enough".
In particular, this definitely does not function off detect evil. That spell returns no reading for an evil creature of 4HD or fewer, but a paladin's smite will still work on such creatures.
There's a different question here which might be relevant -- you might be asking: "how much evil can you do, or intend to do, before your alignment officially changes to 'evil'?". That's an important question but it's very opinion-based. Your DM would have to make a ruling.
Alignments are binary things: you either have it or you don’t. How evil is enough to be Evil is almost-entirely up to the GM,1 but officially every creature in the game has an alignment, which is or is not Evil, and this is an objective, measurable fact of the universe, that can be tested through detect evil or smite evil or any number of other things.
Smite evil, unlike detect evil, has no minimum HD threshold. That HD threshold is not a measure of how evil something is, but rather how powerful it is, and presumably something with less inherent power is harder to detect the evil of (I guess?).