The Hunter Ranger's Giant Killer feature says:
When a Large or larger creature within 5 feet of you hits or misses you with an attack, you can use your reaction to attack that creature immediately after its attack, provided that you can see the creature.
Since it says "you can use your reaction to attack that creature" without specifying that it has to be a melee weapon attack, could you instead use "special melee attacks" like grapples or shoves for this reaction?
Grapple says:
When you want to grab a creature or wrestle with it, you can use the Attack action to make a special melee attack, a grapple.
Shove says:
Using the Attack action, you can make a special melee attack to shove a creature
Since these both normally require taking the Attack action, and Giant Killer doesn't explicitly allow you to grapple or shove, you cannot grapple or shove using Giant Killer.
See this Q&A for a detailed explanation of the difference between an attack and the Attack action: What does upper-case-A-Attack action vs. lower-case-a-attack mean?
Regarding Grappling and Shoving, Medix2 writes:
On grappling and shoving (PHB 195):
[...] you can use the Attack action to make a special melee attack, a grapple. [...]
Using the Attack action, you can make a special melee attack to shove a creature [...]
These are attacks made using the Attack action and are specifically called out in the rules as being "special". They are special for two reasons. First, they do not involve attack rolls, but are still considered attacks. And second, because they replace attacks made as part of the Attack action, and not just any attack. The best example of this is opportunity attacks, which cannot be a grapple or a shove as you are not taking the Attack action but instead a reaction that involves an attack.
