If burning hands is used to set a bush on fire is the damage caused by the burning bush still considered magical damage? What penalties would apply if so?
There is no distinction between "regular" fire damage and "magical" fire damage
PHB 196 states,
Different attacks, damaging spells, and other harmful effects deal different types of damage. Damage types have no rules of their own, but other rules, such as damage resistance, rely on the types. The damage types follow, with examples to help a DM assign a damage type to a new effect. ... Fire. Red dragons breathe fire, and many spells conjure flames to deal fire damage.
Both magical fire spells, such as Fireball, and non-magical sources of fire damage, such as a steam vent, deal fire damage. For example, here's the text for a nonmagical steam vent (OoTA 151):
A hot steam vent erupts beneath a random party member, who must succeed on a DC 12 Dexterity saving throw or take 7 (2d6) fire damage.
And the text for Fire Bolt (PHB 242):
On a hit, the target takes 1d10 fire damage.
See how they're the same? Thus, if a creature has resistance to fire damage, they would have resistance to fire damage from any source, magical or not.
There might be some confusion because certain monsters in the MM are immune to damage from nonmagical weapons. However, the wording is specific that it's bludgeoning, piercing, or slashing damage: (MM 48, demilich)
Damage Immunities necrotic, poison, psychic; bludgeoning, piercing, and slashing from nonmagical weapons
Hence, if you somehow had a flaming, nonmagical sword that dealt fire damage, the demilich would be immune to the slashing damage but not to the fire damage.
If you're designing something for homebrew purposes, then you're going to have to write your own rules that draw a distinction between magic and nonmagic sources.
This would be up to each DM to decide, but my sense is that most would rule that this would NOT be magical damage. Once the bush (as per your example) is on fire, it becomes a normal fire. How much damage it would cause would vary by a) the size of the bush, and b) how close the character is to that bush. But no matter what, it's still just a burning, normal fire with damage. I'd use D4 damage for a regular sized bush fully burning with a character right near to it, and up from there.