It's funny. If you asked any of these AI hyping effective altruists if a calculator 'understands' what numbers mean or if a video game's graphic engine 'understands' what a tree is, they'd obviously say no, but since this chunky excessive calculator's outputs are words instead of numbers or textured polygons suddenly its sapient.
It's the British governments fault for killing Alan Turing before he could scream from the rooftops that the Turing test isn't a measure of intelligence
The fact that it's treated that way is just evidence that none of the AI bros have actually read "Computing Machinery and Intelligence." It's like the first fucking line of the paper.
When I was in college, for one of my classes, one of our assignments was to use a circuit design language to produce a working, Turing-complete computer from basic components like 1-bit registers and half-adders. It really takes the mystique out of computation and you really see just how basic and mechanical computers are at their core: you feed signals into input lines, then the input gets routed, stored, and/or output depending on a handful of deterministic rules. At its core, every computer is doing the same thing that basic virtual computer did, just with more storage, bit width, predefined operations, and fancier ways to render its output. Once you understand that, the idea of a computer "becoming self-aware and altering its programming" is just ludicrous.