Since the GDPR, companies are required to give you a detailed breakdown on why an AI would reject you, if the final decision is on the AI. I'm not sure how many companies are complying though, it's hard to enforce.
The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her. [...]
There is a carve-out if it "is necessary for entering into, or performance of, a contract between the data subject and a data controller", which nobody seems sure what it means, and it has not been tested in court.
I would argue that AI also shouldn’t be allowed to make legally binding decisions, like deciding who to hire. Since a computer can’t be held accountable for its decisions, there’s nothing stopping it from blatantly discriminating.
It should be illegal to use an AI in the hiring process that can't explain its decisions accurately. There's too much of a risk of bias in training data to empower a black box system. ChatGPT can lie, so anything powered by it is out.