How many dice sets would you consider optimum for a group of 4 people (including GM) who want as fast gameplay as possible without dropping more than 50 dollars?
My group, which is that size, finds that in most combats (when dice usage is most intensive) most players will be rolling three yellow dice most of time. (This is for a game that has only been running for half a dozen sessions so far).
You get two of those in a set, so to avoid people having to grab dice off each other mid-round, you should be looking at 1.5 sets per player.
Our group consists of GM + 4 players. We put our Star Wars dice in a pool, held by the GM. They sit in the middle of the table and whenever anyone (GM included) needs to roll, they pull from that pool. This keeps things moving along at a brisk pace. Although there are five dice sets in the pool, because of the group approach to dice it could easily be done with three.