For a quick example of the question posed; say there was a series of spiked mashers randomly coming down from the ceiling during combat between the PCs and monsters. Would the PC's be making dexterity checks or acrobatics checks to dodge the masher plates trying to squish them? Would I need to come up with an attack bonus for the trap and roll against the Reflexes of the characters? And lastly, what about the monsters in the room with the PCs during the hypothetical combat, would I need to calculate the skill modifier (in this case Dex+1/2 Level), use a straight Dex check, or roll against the reflexes with an attack from the traps?
To answer your first question:
- Whenever a skill role doesn't fit, and an ability could feasibly fit, then use the ability check.
The one that comes immediately to mind is that breaking down a door is a strength check rather than Athletics. Most of the time your characters should be rolling skill checks not ability checks. This rewards the skill training and bonus system. Keep in mind that the RC has the updated DC table and that the DMG has outdated advice on how to use the DC table (including adding 5 when rolling skills which is just silly).
Now to address your specific instance.
Use a trap. Traps typically act in one of the following ways:
Like a character in the initiative order, they attack on their turn if a character is in range.
Using opportunity or immediate actions. This means they can either fire once per turn or once per round if their trigger conditions are met.
Traps make attacks, have DCs to detect and disarm and can be attacked ( inanimate objects have AC/Ref/Fort, but not Will and are immune to attacks against will).
If the traps are indiscriminate then they will need to attack the monsters, if the monsters know about the traps and how to avoid them, they might get a bonus. If the traps are controlled by an intelligent source then they might not attack the monsters.
If your traps are outside of combat then you might present them as a skill challenge, in this case have your characters use their skills not their abilities to avoid the challenges.
It's entirely possible that what you're looking for is to set up a "hazard", since it's something that is noticeable and not entirely hidden (sometimes outright overt). In which case you are looking to make an attack and thus use a Defense of the target(s).
Taking a lead from the "Ice Sheet" in DMG 89, you can simply have the mashers make attacks but an acrobatics check can temporarily bypass damage. However, for mashers it could just be an attack against all creatures in that square on its turn, and thus little things like pushes and slides can put a character into an endangered square.