Yes, you go into negative HP with normal mathematics.
In D&D 3.5, damage can take you into negative hit points, with normal mathematics. If you're on 5 HP and you take 10 damage, you are at -5.
If you're reduced to exactly 0 HP, you're Disabled, meaning you can take only a standard action (which deals you a point of damage, reducing you to -1), or a move action. It's actually quite rare to be reduced to exactly zero.
If you're reduced to between -1 HP and -9, you're Dying. You're unconscious, and lose a hit point each round. There's a 10% chance per round to become Stable, meaning you stop losing hit point each round. Healing stabilizes you, and recovers hit points with normal math, i.e. if you're on -5 and get healed for 4, you're still at -1 and unconscious. You can of course still take damage while dying, but most DMs consider this unsporting. Matt Colville has a popular quote from a game he played in: "Oh no. The earth elemental stomps on your head, to make sure you're dead."
If you're reduced to -10 HP or lower, you're dead. No death saves or anything. You can technically go below -10 HP, but there's not much point in measuring it because you're dead anyway. If the example character with 5 HP took 15 or more damage, he'd be straight-up dead. There's a house rule which lets you survive at up to something like negative half your hit point maximum, which is helpful for higher-level characters.