If burning hands is used to set a bush on fire is the damage caused by the burning bush still considered magical damage? What penalties would apply if so?
There is no distinction between "regular" fire damage and "magical" fire damage
PHB 196 states,
Different attacks, damaging spells, and other harmful effects deal different types of damage. Damage types have no rules of their own, but other rules, such as damage resistance, rely on the types. The damage types follow, with examples to help a DM assign a damage type to a new effect. ... Fire. Red dragons breathe fire, and many spells conjure flames to deal fire damage.
Both magical fire spells, such as Fireball, and non-magical sources of fire damage, such as a steam vent, deal fire damage. For example, here's the text for a nonmagical steam vent (OoTA 151):
A hot steam vent erupts beneath a random party member, who must succeed on a DC 12 Dexterity saving throw or take 7 (2d6) fire damage.
And the text for Fire Bolt (PHB 242):
On a hit, the target takes 1d10 fire damage.
See how they're the same? Thus, if a creature has resistance to fire damage, they would have resistance to fire damage from any source, magical or not.
There might be some confusion because certain monsters in the MM are immune to damage from nonmagical weapons. However, the wording is specific that it's bludgeoning, piercing, or slashing damage: (MM 48, demilich)
Damage Immunities necrotic, poison, psychic; bludgeoning, piercing, and slashing from nonmagical weapons
Hence, if you somehow had a flaming, nonmagical sword that dealt fire damage, the demilich would be immune to the slashing damage but not to the fire damage.
If you're designing something for homebrew purposes, then you're going to have to write your own rules that draw a distinction between magic and nonmagic sources.
There is a distinction between magical and nonmagical fire damage.
First, it is explicitly stated in the rules on damage resistance (emphasis mine):
Multiple instances of resistance or vulnerability that affect the same damage type count as only one instance. For example, if a creature has resistance to fire damage as well as resistance to all nonmagical damage, the damage of a nonmagical fire is reduced by half against the creature, not reduced by three-quarters.
Second, it shows up practically in several places. Consider the Armor of Invulnerability:
You have resistance to nonmagical damage while you wear this armor.
While wearing this armor, you would have resistance to fire damage from something like a house fire, but would not have resistance to fire damage from something like a firebolt spell. There is a distinction to be made, as made necessary by this armor, and it is going to be up to the DM what constitutes magical and non magical sources of fire damage. Additionally, the spell gaseous form grants resistance to "nonmagical damage".
This answer goes into greater detail, establishing further a distinction between magical and nonmagical sources of damage.
What about our burning bush?
I will initially say this is up to the DM, because it isn't spelled out clearly, but I will make a case that our bush is probably not burning with magical fire.
To get there we have to ask the question, "What is magical?" This question was asked of Jeremy Crawford often enough that it was answered in some detail in the Sage Advice Compendium. The SAC gives the following definition of magical:
the concentrated magical energy that is contained in a magic item or channeled to create a spell or other focused magical effect.
SAC goes in to give this guidance for determining if something is magical:
Determining whether a game feature is magical is straightforward. Ask yourself these questions about the feature:
- Is it a magic item?
- Is it a spell? Or does it let you create the effects of a spell that’s mentioned in its description?
- Is it a spell attack?
- Is it fueled by the use of spell slots?
- Does its description say it’s magical?
So finally, in the case of our burning bush, I make the case that the fire there is no longer "the concentrated magical energy that is contained in a magic item or channeled to create a spell or other focused magical effect", but rather magical fire just lit the bush on fire that is now burning. After all, the fire on the bush is sourced from the bush itself once the burning hands effect is ended.
So we conclude: fire damage from the bush is non magical.
This would be up to each DM to decide, but my sense is that most would rule that this would NOT be magical damage. Once the bush (as per your example) is on fire, it becomes a normal fire. How much damage it would cause would vary by a) the size of the bush, and b) how close the character is to that bush. But no matter what, it's still just a burning, normal fire with damage. I'd use D4 damage for a regular sized bush fully burning with a character right near to it, and up from there.