The record cut diamond is 105 carats, and is a couple inches long, by about an inch wide, and about 1/3" deep... Records for cut gems in Beryl and Corundum run to about 200-300 carats... That covers most of the precious stones.
Therefore, figure most huge diamonds should be under half a cubic inch; a typical sack can hold thousands. A 2' by 4' sack has a full circumference of 4', and can hold roughly a sphere of .63' radius... roughly a cubic foot... or some 1730 cubic inches.
Huge sapphires, emeralds and rubies, (and several others) run to a couple cubic inches.
A typical sized gem, however, is under 1/16th cubic inch for precious, and under 1 cubic inch for semi-precious.
For simplicity, I'd put the following benchmarks:
Huge 1 - 2 ci
VL: 1/2 - 1 ci
L: 1/4 - 1/2 ci
M: 1/8 - 1/4 ci
S: 1/16 - 1/8 ci
VS: 1/32 - 1/16 ci
Beryl, Aquamarine, Emeralds, Saphires, Rubies, corundum, should run about the same until VL and huge, which should be 1/2- 3/2, and 3/2-3 ci, respectively.
Most semi-precious stones should be fully a size larger than diamonds of comparable relative size...
If you want a truly epic fantasy feel, move the volumes up a step... making huge diamonds some 2-4 ci...