My interpretation: The father is possibly misremembering how quickly he achieved milestones in his early adulthood and he now has unrealistic expectations for his son.
My take is that it's either similar to saying that the dad had to walk uphill both ways or it's just a piece of absurdist humor where the stupidity of the statement is itself the joke.