Keep in mind that there will often be no fruit. Pawpaws spread in monoclonal colonies, and a colony can't pollinate itself. I live surrounded by many pawpaw groves, but only a few actually make fruit. Supposedly you can spray the trees down with fish emulsion to attract the flies that pollinate them, but that still only helps if you have 2 non-clones next to each other
Finding them is pretty easy once you know what to look for, but until you've seen them, it's hard. Easiest time to find them as a novice is in the spring. They have very unique flowers that pop out before there are any leaves. Also, in the late summer/ fall, easiest way to find them is by smell. The fruit is so fragrant, if you know the smell, you can find the tree.
They have those purple flowers that hang downward, right?
I've seen some pawpaw saplings in pots before, but maybe I should have gotten a closer look, haha! I guess that doesn't necessarily translate to how they'll look in the wild. Hopefully I'll have a chance to get the smell down at some point, haha!