If I get up at 4am, I'm going to bed at 8pm. So I'm not saving any time, I'm stealing time from my evening to work in the morning. That's what it ultimately comes down to...stealing personal time to spend on working...
If I'm getting up at 4am on the weekends it means I'm going to be running on ~3 hrs of sleep. Which is fine because at least I'll get a few hours of quiet before everyone else wakes up. (Also why I stay up til 1am)