A dwarf is not commonly seen as a arcane or psionic character. How can I explain in my backstory that my dwarf is able to use psionic powers?
Dwarves aren't commonly seen with psionic powers? Says who? If this is something unique to your setting, you'll have to come up with a story that fits the setting. Otherwise there's no inherent stigma about Dwarves and psionic power use. The psionic power source comes from the far realm and is shaped within one's self: Dwarves are certainly known for their strong self-will.
It's worth noting here that the lore of D&D 4e attributes classes to races by stat, rather than power sources to race by lore. Dwarves are going to be considered commonly anything based primarily on strength, constitution, and wisdom. So they're not going to be commonly thought of as Ardents (cha), Monks (dex), or Psions (int), but Battleminds (con) are totally up their alley.
Evidence of Dwarven Battleminds
number 4, in particular, might be exactly what you're looking for from an game-lore perspective.
Sources for Dwarven Inspiration
Psionic Power page 40 has a little inset called "Psionics? What psionics?" that describes role-playing a character that is unaware of the source of their power, at least at the introductory levels.
The Dark Sun source material makes the psionic power source very common, and dwarves are certainly not exempt. Perhaps you can garner some ideas from their portrayal and adapt them to your setting.
D&D is a very flexible and open world. Come up with anything you want. I role-played a Minotaur Battlemind who came to her power after being a captive thrall of an Illithid doing experiments on the brains of his victims.
If Dwarven Battleminds are rare in your campaign world, come up with some extraordinary story behind your dwarf obtaining his powers. If nothing else, a rich and vivid back story full of unusual and creative ideas could provide fodder for adventuring later on in the campaign.
There is no canon "This is how Dwarves are Psionic and it is the only way" in all of 4E.
So you could steal this instead:
And, of course, the dedication of the Path of Jørn (Plus hanging out in the Underdark with Mind Flayers and other Far Realm beasties) giveth psychic powers. The end.