I don't see that as a problem. They basically only save the phone number, creation date and date of the last connection to the server.
Signal was only able to provide two sets of data: the timestamp of when the account was created and the date of the last connection to the Signal server, as Signal does not store any other information about its users.
source
They do that for easier discovery of existing contacts. Say someone switches from WA to Signal and they want to contact someone. How'd they know their username? Their contact list would be empty and they would uninstall it. Signal would only be for nerds.