Most of this is because, for people who are hiring/interviewing, this is a distraction from the job they were hired to do. Figuring out who to hire isn’t usually one of their core competencies. So they base their decision on superficial bullshit (and then if needed justify their choice later). Often as the job seeker, you’ve learned more about candidate selection than they have, so you’d be better at picking someone than they would.
This doesn't make logical sense. If candidates are studying for what will get them jobs then that wouldn't make them experts in what is needed for the job but the frivolous bullshit that will get them hired.
I think most people who hire people prefer a personal recommendation because they are never trained on how to spot talent. When they can’t take that shortcut, they grasp at straws.
Rarely do you come across someone who actually knows how to pick the best candidate.
Or, picking the best candidate is inherently an impossible task given too little data and too much variability in people's responses and ability to read the interviewer and give them what they want.
In short, the only way to get good at something is to try it repeatedly with feedback. Generalized interviewers / HR perform enough interviews to get better at them, but they don't get meaningful feedback. Whether or not a candidate is actually good for a job often won't be clear for months to years and an HR interviewer is often completely disconnected from that.
Conversely an on-team interviewer might get to see a candidate grow and perform, but simply doesn't perform enough interviews to get good at it. They're too busy working on the team doing stuff and most teams aren't hiring that many people, that often, for them to get enough sample data.
And these forces oppose each other, the more actual task work you do, the less you'll be interviewing others, both because you're busy doing other stuff and because if you're focused in a niche task then you'll have less expertise to interview a broader range of positions. But the more broadly your responsibilities, the less of an expert you are. Same thing with team size, the larger the team, the more hires, but also the more people to do the interviews.
Companies value referrals because the whole interview process is inherently flawed and unfixable.
Hiring for expertise and personailty is a tough set of skills to pair. Reminds me of software I had to work with where you had to have accounting and database skills. Nothing mutually exclusive there, but it's tough to find people with both.
Best you can do is layer it. HR does a quick call to see that you're not an antisocial asshole, maybe something a tad more in depth later, passes to the team's management to go forward.
My last employer did quite well. I sat a 2-hour interview with the whole team. Yes, we got breaks, and as much as anything, that sold me on the job. "The job is very human." And it was! Great company.