I found a bunch of D&D books at a used bookshop, but am now realizing half are 3.0 and half are 3.5... How compatible are these with each other? What are the major rule changes I'll need to watch out for if I'm using these together (or can I not even use them together)?
You can certainly use 3.0e and 3.5e books together. There were many changes between 3.0e and 3.5e, mainly focusing around balance issues. Unfortunately no intentional balancing was done in 3.0, and as such one CR 11 monster would be easy, and another CR 11 monster might be a lethal encounter.
The biggest individual changes are.
- Ranger changes to make them playable.
- Druid changes to make them playable.
- Monsters gain skills and feats.
- Improvements to grid based combat, including the 1.5 diagonal rule (which was removed again in 4e).
- Changes to many spells. For example TimeStop can only be used to effect the caster. Before it could be used to destroy enemies entirely.
- No class-specific skills. Skills that are marked "exclusive" in 3.0 are "trained only" cross-class skills in 3.5.
The official change guides can be found here.
I'll try to give a response that's not technical but rather revolves around why you asked this question.
Apart from the slightly different rules in pretty much everything, from grappling to sundering to how damage reduction works to the levels at which you get some powers and some spells, 3.5 is sort of a big errata.
The main changes were fixes to spells such as harm (harms an enemy by touching him, leaving him at just 1d4 HP, with no way to avoid it), haste (too strong effect for casters, too strong defensive bonus), polymorph (no limit on the powerfulness of monsters you can become, only size matters), several "all day long" spells reduced to "choose which to cast during combat."
Most 3.0 material that has not been revised could be ported as is and not cause problems, but much of the rest is just broken under 3.5 rules. I'd suggest that, if a 3.x experience is what you are looking for, you play 3.5 or even Pathfinder RPG. Since you're buying 3.5 material you can read most of the rules in the System Reference Document (SRD).
You'll notice much has changed (for example, how the size of a creature determines its dimensions on the battle grid) and 3.0 material might be awkward to use unless you have a good knowledge of both systems. Such as the knowledge of a 3.0 player who moved to 3.5 over time.
Also, remember that even in 3.5 a lot of revisions have been made. The same spell could have been published in different books and be therefore unbalanced if the book you own is not the primary source for that info (as defined in the numerous errata). Unbalanced seldom means unplayable though, but navigating your way among the books is hard if you're starting from scratch.
Anyway, the SRD and any 3.x edition's PHB and DMG are enough to play the game (the DMG contains wealth by level tables for PCs and NPCs, the PHB has the XP table and everything about leveling up).
C. Ross hit the big ones. I'd add:
- Skill changes -- some skills were renamed, some were removed, a couple were added
Just because that's a big enough change so that old material might be confusing.
There's also a fan-generated change list by Steven Cooper. The original site is down, but the above link uses the Wayback Machine, so it'll work even though it's slow.
Answering the question in a different sense, D&D 3.5 changed the emphasis from flavour to mechanics. D&D 3e essentially took AD&D classes, races, themes, spells and such and converted them to the new d20 system. D&D 3.5 took the d20 system as a base and built the races, themes, classes and spells around them. Significant changes like Rangers not getting dual wielding at level 1 (which was thematic from AD&D, but led to everyone multiclassing Ranger if they were minmaxing), and the nerfing of Haste were due to this. As a result, D&D 3e and D&D 3.5 have a significantly different feel, even though some of the answers would suggest that from 3e to 3.5 and from 3.5 to Pathfinder were similar changes, 3e to 3.5 was a massive change, to a certain degree even moreso than AD&D to 3e.
One of the things not mentioned yet is how damage reduction worked. Specifically in 3e entries, you'll see things like 'damage reduction 10/+2'. That meant that it had damage reduction 10, and a +2 or greater magic weapon could bypass it.
In 3.5, it'd just be 'damage reduction 10/magic'. Any magic weapon will bypass that. But then you'll see something like 'damage reduction 10/slashing and good', which means the weapon has to be both a slashing weapon AND good aligned.
They also trimmed down haste a lot, resurrections changed and Heal has been "NERF"ed considerably (it was such pleasure filling to the top the barbarian 18-wheeler-sized tank of hitpoints with just one spell. No more in 3.5).
There's one more change. The Beast and Shape changer types have been removed. Beasts are simply understood to be animals but unfortunately shape changers have their type turned into a subtype and gain an entirely new type which seems to be unique for each creature.
This might not seem problematic at first, however, it might be difficult to locate conversion guides for every book with monsters, meaning every former shape changer type monster has to be DM judged on a case by case basis which is time consuming at the very least.
My personal "rule of thumb" and it's the same for every group that I know in my region is: - Newest rules takes precedense!
Let me elaborate on that a little. We consider D&D 3.0, D&D 3.5 and D&D 3.75 (which is Pathfinder) to be the one and the same system. In fact creators of those systems treat them that way and differences between them are more of a "improve, balance and specify" than "create something different". All those systems are officialy announced as compatible and if You're ever having dilemma, there are official "upgrade FAQ's and GUIDES" available.
Since most of content is being updated between those editions it's easiest and most balanced to take newest versions of things everytime. So for example if some prestige class have only 3 Edition version we use that. If it has both 3 edition and 3.5 edition version we use 3.5 version. If it has both Pathfinder, 3.5 and (not necessarily) 3.0 edition version we use Pathfinder. Mind that some prestige classes (since that's what my example is based on) are turned to archetypes in Pathfinder. The same thing goes for spells, feats, etc.
Of course if You aim to perfectly balance Your game (which is impossible task with different tiers of classes, inequality of builds, etc. and I believe You shouldn't even try since it's not worth all the hassle) You'll need to make some homebrew changes, be we're rarely concerned of that.
The main profit of this approach is You get so gigantic amount of content to choose from that You can create absolutely anything with, both as player and DM, without resorting to house-ruling anything at all. This makes everyone's life easier and let's everyone focus on an actual game more than arguing about rules. Manualy picking what is and what isn't allowed would destroy all that...