They actually do have the Holy Avenger (albeit differently) in 4e!
Holy Avenger Level 25+ Rare
The most prized weapon of any paladin.
Lvl 25 +5 625,000 gp Lvl
30 +6 3,125,000 gp
Weapon: Axe, hammer or heavy blade
Enhancement Bonus: attack rolls and damage rolls
Critical: +1d6 radiant damage per plus, and you can spend a healing
surge
Special: A holy avenger can be used as a holy symbol. It adds its
enhancement bonus to attack rolls and damage rolls and the extra
damage granted by its property (if applicable) when used in this
manner. You do not gain your weapon proficiency bonus to an attack
roll when using a holy avenger as an implement. Property
Your radiant powers deal 1d10 extra radiant damage when used through
the weapon. Power Daily (Minor Action)
You and each ally within 10 squares of you gain a +5 power bonus to
fortitude, Reflex, and Will until the end of your next turn.
I'd also suggest Vigilant Blade for the holy concentrated weapon feel with more damage focus available at a lower level.
Vigilant Blade Level 4+ Uncommon
Set with the symbol of an upright gauntlet with an unblinking eye in
its palm, this blade bursts into burning white light in the hands of
one sworn to bring justice to the wicked.
Lvl 4 +1 840 gp Lvl
19 +4 105,000 gp Lvl 9 +2 4,200 gp Lvl 24 +5 525,000 gp Lvl
14 +3 21,000 gp Lvl 29 +6 2,625,000 gp
Weapon: Heavy blade or light blade
Enhancement Bonus: attack rolls and damage rolls
Critical: +1d8 fire and radiant damage per plus Property
Paladins and avengers can use this weapon as an implement for paladin
or avenger powers and paladin or avenger paragon path powers.
Power (Minor Action)
The weapon casts bright light out to 4 squares until you end this
effect as a minor action.
Power Encounter (Immediate Reaction)
Trigger: You use this power when an ally adjacent to you is attacked
by an enemy adjacent to you.
Effect: You switch places with the ally, and the target grants combat
advantage to you until the end of your next turn.