\$\DeclareMathOperator{\vs}{vs.}\$Here is a table of how much better (the ratio of the success rate with the feat to the success rate without it, expressed as a percentage—100% means the feat does not change your success rate) each feat is. \$SR - CL\$ is the target spell resistance minus your caster level (before either feat), that is, the roll you need on the d20 to succeed.
SR − CL |
Arcane Mastery |
Spell Penetration |
≥23 |
N/A¹ |
N/A² |
22 |
N/A¹ |
∞ (10% vs. 0%) |
21 |
N/A¹ |
∞ (5% vs. 0%) |
20 |
N/A¹ |
300% |
19 |
N/A¹ |
200% |
18 |
N/A¹ |
167% |
17 |
N/A¹ |
150% |
16 |
N/A¹ |
140% |
15 |
N/A¹ |
133% |
14 |
N/A¹ |
128% |
13 |
N/A¹ |
125% |
12 |
N/A¹ |
122% |
11 |
N/A¹ |
120% |
10 |
200%* |
118% |
9 |
181%* |
117% |
8 |
154%* |
115% |
7 |
143%* |
114% |
6 |
133%* |
113% |
5 |
125%* |
113% |
4 |
118%* |
112% |
3 |
111%* |
111%* |
2 |
105%* |
105%* |
≤1 |
100%† |
100%† |
For SR − CL greater than 10, you cannot use Arcane Mastery (or, doing so guarantees failure), so it has no effect on your success rate.
For SR − CL greater than or equal to 23, you cannot succeed even with Spell Penetration, so the feat has no effect on your success rate.
* Guaranteed success with the feat.
† Guaranteed success even without the feat.
So Spell Penetration helps in a lot more situations, and helps more in its best use-cases. But Arcane Mastery guarantees success against far more creatures, and creatures with SR high enough to really put Spell Penetration to good use, are really creatures you really don’t want to be checking spell resistance against. Even though Spell Penetration literally gives an infinitely-greater chance of success if the SR was 21 or 22 higher than your CL, you are still left with only a 10% or 5% chance of actually beating the SR with Spell Penetration. Something with SR that high is far too dangerous to take odds like that—you need to use SR: No spells, or probably better, running away.
Which is really the big problem with either approach to SR: you are investing a huge amount (a feat) into beating SR, when you really want to not have to. There are a lot of alternatives for spellcasters to avoid having to actually roll against SR at all, and they are often very good.
And if you really want to use an SR: Yes spell in such a situation, these feats aren’t the best answer anyway. That would be assay spell resistance from Spell Compendium—that applies a whopping −10 penalty to a target’s SR. If you have SR: Yes spells that you want to be able to use, prepare a few copies of assay spell resistance, and be done with it.
All that said, Arcane Mastery isn’t a bad feat for other purposes, perhaps most notably dispel magic. If you must pick one of them, pick that one just because it does more than just SR.