Up to your GM
Assuming you have free picks of language left, the Player's Handbook tells you to choose any languages you can pick yourself from the Standard Languages --- Common, Dwarvish, Elvish, Giant, Gnomish, Goblin, Halfling or Orc --- or to pick one that's common in the campaign you're playing. However, it also says the following:
With your DM's permission, you can instead choose a language from the Exotic Languages table or a secret language, such as thieves' cant or the tongue of druids.
Therefore, all you need to do is convince your GM to accept the choice of language. The choice of languages is usually not very important in a game, so typically requests like this are easy to accommodate, but of course your GM's game might play up their importance, in which case they're the best person to determine whether the choice of language is bad for the game or not.
If you have no free picks of language left, prepare to argue harder, as you're essentially asking for a free (if slight) bonus to your character's abilities. It might help to point out that Abyssal is, in practice, of relatively little use as the bulk of its speakers are demons who prefer destruction over discussion.