There's no reason to think that 'number of damage dice' refers to the number of sets of dice
Nothing in the rules says that. In fact, even the fact that you roll a weapon's damage dice rather than it's damage die implies that the weapons may well have more than one such die to be rolled. The number of dice rolled for a Greataxe is one d12, that is, you roll one twelve-sided die. A Greatsword rolls two d6, or two six-sided dice. The intro chapter covers this, saying:
Whenever a roll is required, the roll is noted as “d#,”
with the “#” representing the number of sides on the
die. If you need to roll multiple dice of the same type,
there will be a number before the “d.” For example, if you
are required to roll 4d6, you should roll four six-sided
dice and add the results together.
(note for non-english speakers: the plural of 'die' is 'dice')
if the feat was intended to use the same vital strike multiplier, it could easily have said so, or even said 'the number of times your weapon dice are rolled' instead of 'the number of weapon dice rolled for the feat'. The latter is about as explicit as the could possibly be that you multiply the damage done by the total number of weapon dice rolled for the feat.
So the multiplier clearly goes up the really-big way and not the looks-really-big-but-is-actually-100%-a-trap way.
Yep, that's crazy
It's quite a lot of damage if you go about it right. Specifically, you multiply the applicable bonuses by 64 in the most extreme case.
It's not that crazy, though
30,000 damage on a single non-critical hit? that would imply your damage modifiers summed to 30,000/64, or ~469. I'm pretty sure you can't even get that high of a bonus. Even your 10,000 damage estimate is far overblown. We're probably looking at something more like 36*64=2304 damage for a very-damage-focused mid-op high-level character.
So, you get to, on a hit, do ~2304 damage. That's a lot, enough to one shot Cthulhu. You don't really need to ever do more damage than that. But what does this mean? Generally, this means you get to use your standard action to kill one creature with a successful attack roll v.s. AC. That's effectively a save-or-die with an unusual targeting statistic (AC), which is pretty cool. Furthermore, most save-or-die's in Pathfinder require two saves rather than one, and most of them are spells.
Nonetheless, this is an ability typically only usable against a single target each round (although beware X-Lasers, shotguns, and similar), with a very large number of ways to render it completely impotent. It's not so much broken as it is highlighting the ways that hp damage becomes sort-of irrelevant at higher levels.
You can change it if you want
X2, X3, X4 are certainly more consistent with the effects of the prerequisite feats. Making this change punishes mundane characters to the advantage of casting classes, which is consistent with the Pathfinder design eidos. Banning or nerfing a single specific feat is very unlikely to change the overall way the system plays, while making a feat more powerful does risk significant and problematic changes, from a theoretical standpoint. So if you do decide to make it work this way it will look and feel fine, basically.
You don't need to change it, though.
The mythic optional rules are all about the players having overtly super-powered versions of normal characters. Blowing up basically everything ever with a giant oversized doomsday attack is very much consistent with that design philosophy. Even if mundane and melee characters aren't supposed to get anything nice ever, they feature largely in the faux-mythological material that 3.X is descendant from. Allowing a player, as a very high level ability, to make use of this build (Greater Vital Strike requires 16 BAB) is not a significant problem.
Do look out for Improved Mythic Vital Strike, though, which allows the damage to rocket towards your initial estimates and makes this sort of problem available at 11th level.
It seems like you might be having touble understanding how big numbers of weapon damage could not be a big deal at higher levels. Lets look at what sorts of defences high-powered 17th level characters typically have, and whether an infinite-damage melee vital strike would be enough to overcome them.
Our first test cases are going to be taken straight from the SRD so there's no building involved and it's clearer that these sorts of things ought to be expected:
Bakekujira:
This undead whale's long-range-strike capability allows it to close with you first, despite its massive size, ensuring that you begin your turn within its Undead Parasites effect. This means that you must make a DC 28 Will Save or be nauseated and thus unable to attack (or spend some channel energy, if you have that ability, and have the Whale fail a save which it needs to roll a 1 to fail), and then hit AC 32. If you succeed at both checks, you kill the whale, if you fail you do nothing with your turn except possibly cause yourself more damage. You have a reasonable shot at killing this thing each round, though it will do a lot of damage to you first.
Bandersnatch:
The Bandersnatch confuses you (DC 29 Fort save to negate), possibly resulting in you OHKOing your party members. Particularly beware of the Confounding Bandersnatch, which has additional methods of confusion (DC 29 Fort save injury poison). Alas, the Frumious Bandersnatch is no more a threat to you than a regular one. In addition to confusion, the Bandersnatch is quite capable of grappling you with its grab special ability, rendering you unable to use your giant pile of feats. It's also probably somewhat faster than you, gets to do a bunch of stuff as swift actions that are normally standard actions, and generally can use the saved time to out-range you. It possesses okayish ranged attacks that inflict the sickened condition on its targets, in addition to the ranged swift action confusion. It can charge through difficult terrain, and ignore most applyable conditions with ease. You're pretty much dead. Ready an action to strike it when it appears, and hope you pass your will save(s) and the rest of your party can force it to close.
Phasmadaemon
Uses illusions and misdirection to evade your attacks. Also grapples, which negates your abilities. Also can fly, which renders you irrelevant unless you can, too. Also has its own suite of more traditional OHKOs.
Maralith
Grapples, rending your abilities irrelevant. Constricts, rending anyone's abilities irrelevant with a DC 25 Fort save to negate. Gets so many attacks each round. Could try to disarm you with each. Has the hp to take hits from your unarmed strikes, assuming that's not your main weapon (you do something like 2d6X4+strX8 damage, and provoke AoO with that). Can teleport at-will. Can create illusions. Can see through your illusions. Can force choke you out with telekinesis. Can fly. Has an army. 20% chance you're fighting two of these. Mythic Maraliths can Mythic Blade Barrier, which is an immediate action. AC 32, but 36 with Blade Barrier's cover effect up. You can't really do too much here, armies are a major weakness of yours, as are grapple attempts (let alone ranged insta-gib grapples).
Lilitu:
Has 9 lives. As in, you can kill this thing 8 times in a row without it doing anything. That's... a pretty rediculously good defense. Also can use binding and disguise and teleport and dominate monster and all that good stuff. Basically, you are so, so screwed, but so is everyone else in your party. Hope you brought an optimized T1 full-caster... or two...
Rune Giant
AC 30 and you are blinded so there's also a 50% miss chance. Has Dominate Person. You can take these guys handily-- the defenses are substantial but just not enough to reliably stop your OHKO and even if you fail, dominating you means your party has yet another full round to beat on the enemy or free you. Plus mind control is pretty easy to block.
Formian Queen:
Has an army, also mind control, also Magic Jar at will and teleport. Mostly the army is the problem. If you can get past that, her only really recourse is to run away. She can probably escape you but that's hardly a victory, and being able to force her away from whatever you want is basically winning. If you really care enough you can probably run her to ground eventually, you'd just have to be pretty determined.
Immortal Ichor:
Oh, no! Swarms! You can't fight swarms, they're just straight-up immune to you. Oh well, guess you sit this one out.
Irminsul:
...This should not be CR 17. You can kill this less well than most people, since it can actually attack you back, but a party of well-built level 5 characters could take it out if the felt like it. Heck, a party of level one characters can kill this thing if you pick the right ones. But it is published as CR 17 and Mythic and you can kill it real easy.
Kaminari: It flies, it can merge with its storm cloud, rendering you unable to harm it, you actually can't approach it unless you have 15 ft reach because it has the Stand Still feat, which prevents you from being able to move close enough to hit it with most melee weapons. It can shoot lightning bolts from 1200 ft away, including 1/day an empowered chain lightning for 30d6 damage to everyone. It's invisible, and can knock out magical methods of flight with greater dispel magic. Mostly, it's impressive offensive capabilities are intended to double as defensive deterrent, but in this case you also just straight up can't hurt it without significant support (and even then it's invisible and has 32 AC and such) cause you're a melee character, and you generally can't get support for this one because greater dispel magic will knock out most of your important augmentations as well as your buddies'. Definitely a hard fight and not something you're likely to be a major factor in.
Khala:
You're actually a pretty solid combatant against most dragons, but this one's secondarily a grapple monster so you're just really good rather than a shoo-in. You can probably kill it, assuming you have your party for backup.
Termagant:
It's a grapple monster with a save-or-suck gaze attack targeting Will. It's got a number of methods to neutralize you, though, obviously, if you get a hit off it's dead. It's got a negligible poison effect thingy that really isn't a problem for anyone at level 17 and especially not you.
Lusca:
A faster, more dangerous grapple monster with a whole lot of grapples and not a whole lot of you getting out of the grapples. Not a good fight for you.
Keketar:
You can literally do nothing against this creature. It has greater teleport at will, and anyone teleporting near it loses at least one turn, no save. Also it has lots of other abilities, but that's enough to make sure that you cannot close with it on your own. You would need someone to teleport you 40 ft away from it with your initiative next coming before the Protean's and hope you weren't teleporting directly into a Prismatic Sphere. Even then it can grapple you with its AoO, is probably hidden requiring Perception to locate, and has 32 AC. Offensively, it can kill you pretty easily assuming its quickened confusion doesn't have you fighting your own party. Also it gets PaO 3/day, which is one of the most powerful spells in the game and allows it to ensure you are fighting whatever it wants as well as it. Another one for the significantly more optimized parties.
Shinigami:
You can beat this. It relies exclusively on stacking negative levels to kill stuff, and only does like 8 negative levels a round, tops, so if you hit its 31 AC within a couple rounds you're good. It can fly, though.
Thrasfyr:
You can beat this, but it's not intended as a solo monster but rather as a minion of something more dangerous. Fights that aren't against solo monsters aren't your strong suit. You can beat this, though!
Tunche:
You can beat this, probably. It's got nausea poison and tree shape, mostly.
Winterwight:
You can beat this. It's got nothing, except the possibility that you'll slip because ice is slippery. Since it has no realistic way of harming you, you can just get back up and continue trying to hit its 32 AC, even if that does happen.
So, there's a bunch of fights where you can contribute to your party and a coordinated effort around you would lead you to probable victory, a number of fights where you're utterly useless, and a number of fights you could win even on your own maybe. That's not an 'overpowered' character, that's actually surprisingly balanced for high-level Pathfinder play. You aren't the wimpy ne'erdowell of the typical melee combatant, and you aren't the omniscient omnipotent deity of the typical magic-user. The only real problem is that you'll likely outshine or be overshadowed by most straight base-class characters; there isn't a whole lot in your power bracket at level 17.
Now, let's compare your thing to typical high-level PC abilities:
You: one-roll save or die vs standard AC requiring melee range.
Time Stop spam: victory with no save, opposable only be specific countermeasures. Way more powerful than your thing.
Dominate Monster: one-roll save or die comparable to your attack, but strictly weaker as immunity to mind-affecting spells is more common than immunity to weapon damage, especially at high levels. Furthermore, this requires a Full-Round Action rather than a Standard Action. However, this nets you a semi-permanent increase in power since you get the defeated creature rather than just killing it and also it prevents the use of True Resurrection spam as a defense.
Wish/Miracle spam: Much better defensively, Wish kills enemies by dropping them in hostile places, with a Will save allowed, and with Spell Resistance allowed twice. However, it has no range. Like, it works across planes and without line of effect. Mostly, it's just different.
Druidic summoning spam: Druids get armies of increasingly powerful free-willed beings. It's hard to compare their abilities to yours, but it generally seems like the army's versatility and resilience is superior to your single-target effect.
Deatch Clutch spam: hp-limited one-roll save-or-die, but always makes the opponent staggered and might deal serious Con damage against opponents with too much hp. The effect is a little less reliable than yours, but the 60 ft range is a lot better than melee, so we'll call it even.
Imprisonment spam: single Will save at -4 or die-and-don't-come-back, melee range standard action. This is basically the same thing you do only slightly better because casters have to be better at everything :/
Weird spam: Will save or die (succeeding the Fort save leaves you stunned, which is basically dead), AoE for any number of creatures within 30 ft of each other. At high levels, where everything has telepathy, this becomes a contest of wills with any creature that makes its save-- whoever fails the will save first dies. Unfortunately, is a mind-affecting illusion fear emotion effect, and thus vulnerable to almost every high-level mental defense ever. Still generally better than your ability, though.
And so we see that good 9th level offensive spells are generally better than your ability, but not by terribly much most of the time. Now lets look at what it costs to become completely immune to your ability:
Swarm Skin is a 6th level spell that allows you to become swarms that are immune to weapon damage
Shadow Body is a 5th-7th level spell that grants you incorporeality.
So people looking to become immune to your attack can do so between levels 9 and 13, generally, and at 17th level, when you get the attack, PC-equivalent enemies can easily render themselves immune.
