Take the ASI, unless the campaign is overrun by resistant enemies
It all will come down to how often you face fire resistant opponents. If it is really a lot then maybe Elemental Adept (EA) might be worth it. Without fire resistance running rampant, i.e. for just the occasional enemy with it, EA is not worth it vs an ASI.
Fire damage spells
EA's plain improvemement on the damage from fireball is 0.17 per each d6 (as treating a 1 like a 2 will improve the average damage from 3.5 to 3.67). That means, absent fire resistance, you gain a total of 1.3 extra damage from a 8d6 fireball.
The ASI increases the chance that the opponents do not save against fireball by 5%. With an expected average damage of 28 for a failed save vs. 14 for a made save, this improves your expected fireball damage by 0.7 points.
So its 0.6 damage in favor of EA per fireball. That is not relevant.
On Fire bolt, the improvement is 0.1 points for EA, vs 0.275 from ASI, so a without resistance, the ASI outperforms EA when it comes to damage, but also not so much that it matters.
On Burning Hands the pattern is the same as for fireball, just scaled down. On Scorching Ray, EA gives you 1 point of extra damage, just like the ASI does by increasing your hit chance.
Once you get to add your ability bonus to damage on level 6, the ASI outperforms the fire damage increase for EA on all these spells.
Other benefits of ASI
An ASI improvment will also increase all your Charisma based skills, and your Charisma save, and your spell DC for all your other spells, not just the fire-based ones.
I do not know how your build otherwise looks like, it seems quite focused on dealing fire damage. Typically it can be useful to mix the different kinds of offensive spells and also include ones that remove a single strong opponent from combat.
In any case, the versatilty of the ASI is more useful unless you are swarmed by fire resistant opponents.