The only reason why xz got exposed was because someone noticed SSH was a bit slower and decided to take it to their own hands to investigate. It’s possible this backdoor would go unnoticed for far longer if the attacker didn’t make this slight oversight.
So it might be that there have been other, successful attacks before. It’s just that this one is the one that got exposed.
SSHing into my less powerful machines takes a good few seconds, so I'm not sure if I'd notice an extra 500ms. For the more powerful ones that are basically instant it would be much more noticeable.