There are apps made for linux that don't work with android, and there are apps made for android that don't work with linux. That's enough for me to consider them different
Also android just doesn't use the basic mainline kernel which is what most people want when they say "linux phone"
glibc is key here, it's what most linux distros use. One of Google's vendor-lock moves was to start using their own libc implementation, making it incompatible with everything else.
It is also a highly modified kernel, extremely reduced. They do all filesystem stuff in userspace for example, which is pretty cool. And they add a ton of garbage out of tree drivers.