No touching or stealing does not end the spell, as there is no attack roll involved (thus no attack), it is not a grapple or shove, which are special attacks requiring the Attack action, and you have not cast a spell.
The conditions for breaking invisibility are succinct. 5e attempts to have spells do exactly what they say they do, no more, no less, and with no hidden presumptions. You can tap someone on the shoulder or pick their pocket without fear of losing your invisibility.
This is supported in many posts of the lead designer, Jeremy Crawford and succinctly stated here:
A spell's text details the spell's effects—the only thing the spell
does. Any additional effects are up to the DM
Beware of claims that a rule does something mentioned nowhere in that
rule or elsewhere in the core books. There aren't secret rules.