Powerful Build alone: no
The measure of how much effort it takes to use a weapon (whether the weapon is designated as a light, one-handed, or two-handed weapon for a particular wielder) is altered by one step for each size category of difference between the wielder’s size and the size of the creature for which the weapon was designed. If a weapon’s designation would be changed to something other than light, one-handed, or two-handed by this alteration, the creature can’t wield the weapon at all.
So a goliath can wield a Large weapon without problems. A Huge weapon would go up an effort category, and a Gargantuan weapon another: thus a Gargantuan light weapon becomes a two-handed weapon for a Large creature, or for a goliath. But for a Colossal weapon, even a light weapon would go beyond being two-handed, and that means that the goliath cannot use it, no matter how large the penalty. We therefore need something extra.
Feats and magic items: maybe, but only Colossal one-handed weapons as two-handers, and that’s pointless
Powerful Build does not stack with Monkey Grip or strongarm bracers, so those are out. Wield Oversize Weapon does seem to work, strict-RAW anyway, but the fact that Wield Oversize Weapon is restricted from working with Monkey Grip means it probably should also be restricted from Powerful Build (which didn’t exist at the time Wield Oversize Weapon was written).
Even if it does work, turning a Colossal one-handed weapon into a Large two-handed weapon and taking a −4 penalty is pointless; the damage of a Colossal one-handed weapon is generally identical to the damage of a Gargantuan two-handed weapon, so you could just take Wield Oversize Weapon and use a Gargantuan two-hander as if it were Large, take a smaller penalty, and get about the same damage.
Real magic: yes, but only by getting larger and ceasing to be Medium.
The other solution, and a much better one, is to just physically get larger yourself. A fighter has almost no way to do that, but a psychic warrior has a very good one, in the form of expansion. With a fair amount of augmentation, expansion can increase one two size categories, so that the goliath is, himself, Huge, and can wield Gargantuan weapons without penalty. That means he can wield Colossal weapons by allowing them to go up an effort category and taking a −2 attack bonus.
To go even farther, giant size is a 7th-level wu jen spell; it is personal-only, which makes it difficult to get on anyone who wants to be physically fighting, but a wu jen gish build could enable it on our goliath. It also takes 1 round to cast, which is awful, and lasts only 1 minute, which is atrocious, but the effect is quite significant. At caster level 16th-18th, giant size makes you Gargantuan, which for a goliath allows the wielding of Colossal weapons without penalty. And at 19th caster level and higher, it actually makes you Colossal, which actually makes Powerful Build kind of pointless.
Result: a small increase in damage, relative to the large investment
The effect of all this investment (LA +1 for Powerful Build, −2 or −4 attack penalties, lots of power points, the trouble of making oneself a gish and using a 1-round casting time, 1-minute duration spell), at least so far as the weapon is concerned, is at best (greatsword) a paltry 6d6 damage, an expected increase of only 21 damage. As always, increasing the size of your weapon is a terrible way to improve your damage. At least the expansion and giant size options also increase your space and reach; those are valuable properties. But merely having a larger weapon does nothing for your reach, the only thing it improves is your damage, and that slowly.
Recommendation
Being a Medium creature that wields a Colossal weapon is conceivably possible with Wield Oversize Weapon, but you take large penalties and receive a very small damage bonus for doing so. It also requires an LA +1 race and several feats. This is not a good trade.
On the other hand, actually becoming physically larger yourself results in improved space and reach, which are very valuable. Increasing your actual size is also, ironically, easier than wielding larger weapons than you’re supposed to, so you get the (still small) damage bonus cheaper than you would otherwise.
By far your best bet here is to just be a psychic warrior; they’re a competent and fun class, and expansion is convenient and effective. You can be a goliath if you insist, but Powerful Build’s effect on weapon damage is minimal; Powerful Build is really only good for combat maneuvers, particularly bull rush, grapple, and trip.
Note, however, that even goliath psychic warriors cap out at using Gargantuan weapons without penalty, not quite reaching Colossal. To actually get Colossal without penalty, you’ll need either Wield Oversize Weapons (and a DM who allows it to stack with Powerful Build), or to go with wu jen and giant size – and the wu jen can actually get to Colossal without Powerful Build. However, the drawbacks of each of these approaches (just taking the penalty, spending feats, being a wu jen) are not even remotely compensated for by the small improvement in damage you get for going from Gargantuan to Colossal. Actually, for that matter, the LA +1 of goliaths is not worth it; sticking with a regular-sized weapon (i.e. Medium normally, Huge after expansion) is better than any of these options.