I always keep my phone in my front pocket and have never had a problem biking. Are you actively buying pants with tiny pockets? I only buy second hand clothes and have a few "female pants" with pockets that don't fully fit huge phones and even with them it's not a problem.
Maybe it's improving. There are a few less famous brands making some small android phones. In a recent thread, someone mentioned unihertz, and I added cubot.
People keep saying those phones aren't small without saying what size or current phone is small enough. The fact the base galaxy phone has shrunk from 6.2-6.3 inches is impressive enough IMO.