Eh I don't think it's malicious in nature but can't prove it either is or isn't. They might be doing more analysis on some outbound links or users for something or just A/B testing some additional methods for gathering more data. Unsure. But I wouldn't immediately jump to intentional.
i think you should read the replies to the hackernews post, there's some interesting discussion there with more nuance. particularly the thread started by user "kens"
yeah, but that applies to a problem where you don't have any additional information. if you do have background information of any kind, you would be wise to take them into account.
The fact that specific domains and user agents are effected by this says otherwise. Take a look in the link for people discussing curling the urls and their findings.