Just wonder what if my mail server went offline for some periods, and the sending party couldn't deliver.
Will there be any consequences except I don't get the mail? I tried searching but they all in the perspective of a sender and get a bounce, rather the other way around.
I've been self-hosting Postfix for several years and it's not difficult, if you're absolutely confident what you do. I don't recommend it if you don't know basic behaviors and internals of SMTP and relaying. Also you need to know how to secure your server so you don't get spammed a lot and getting hammered with brute force attacks.
From time to time you need to react to delivery problems. Most interesting one is perhaps Microsoft, which you need to ask to whitelist your server or your email won't be accepted.