How do we know if there aren't a bunch of more undetected backdoors?
How do we know if there aren't a bunch of more undetected backdoors?
I have been thinking about self-hosting my personal photos on my linux server. After the recent backdoor was detected I'm more hesitant to do so especially because i'm no security expert and don't have the time and knowledge to audit my server. All I've done so far is disabling password logins and changing the ssh port. I'm wondering if there are more backdoors and if new ones are made I can't respond in time. Appreciate your thoughts on this for an ordinary user.
"We don't" is the short answer. It's unfortunate, but true.
How do you know there isn't a logic bug that spills server secrets through an uninitialized buffer? How do you know there isn't an enterprise login token signing key that accidentally works for any account in-or-out of that enterprise (hard mode: logging costs more than your org makes all year)? How do you know that your processor doesn't leak information across security contexts? How do you know that your NAS appliance doesn't have a master login?
This was a really, really close one that was averted by two things. A total fucking nerd looked way too hard into a trivial performance problem, and saw something a bit hinky. And, just as importantly, the systemd devs had no idea that anything was going on, but somebody got an itchy feeling about the size of systemd's dependencies and decided to clean it up. This completely blew up the attacker's timetable. Jia Tan had to ship too fast, with code that wasn't quite bulletproof (5.6.0 is what was detected, 5.6.1 would have gotten away with it).
In the coming weeks, you will know if this attacker recycled any techniques in other attacks. People have furiously ripped this attack apart, and are on the hunt for anything else like it out there. If Jia has other naughty projects out here and didn't make them 100% from scratch, everything is going to get burned.
I think the best assurance is - even spies have to obey certain realities about what they do. Developing this backdoor costs money and manpower (but we don't care about the money, we can just print more lol). If you're a spy, you want to know somebody else's secrets. But what you really want, what makes those secrets really valuable, is if the other guy thinks that their secret is still a secret. You can use this tool too much, and at some point it's going to "break". It's going to get caught in the act, or somebody is going to connect enough dots to realize that their software is acting wrong, or some other spying-operational failure. Unlike any other piece of software, this espionage software wears out. If you keep on using it until it "breaks", you don't just lose the ability to steal future secrets. Anybody that you already stole secrets from gets to find out that "their secrets are no longer secret", too.
Anyways, I think that the "I know, and you don't know that I know" aspect of espionage is one of those things that makes spooks, even when they have a God Exploit, be very cautious about where they use it. So, this isn't the sort of thing that you're likely to see.
What you will see is the "commercial" world of cyberattacks, which is just an endless deluge of cryptolockers until the end of time.
Self hosting personal photos doesn't generally require opening anything up to the internet, so most backdoors would not be accessible by anyone but you.
Or someone who has penetrated your network.
Of which the chances are slim to none for 99% of people simply because they aren't interesting enough to be a target beyond phishing, scans, and broad attacks.
Cheeky answer:
Actual answer:
Theoretically anyway, open source software's guarantee of "no backdoor" is that the code is auditable, and you could study it and know if it has any holes and where. Of course, that presumes that you have the knowledge AND time to actually go and study thousands of lines of code. Unrealistic.
Slightly less guaranteed but still good enough to calm my mind, is the idea that there is a whole-ass community of people who do know their shit and who are constantly checking this.Do note that like. Closed source software is known to be backdoored, only, the backdoors are mostly meant for either the owners of the software (check the fine print folks) or worse, the governments.
The biggest thing that you should note is that: It is unlikely that you (or I or most of the people here) are interesting enough that anyone will actually exploit those vulnerabilities to personally fuck you over. Your photos aren't interesting enough except as part of a mass database (which is why Google/Facebook want them). Same for your personal work data and shit.
Unless those backdoors could be used to turn your machine into a zombie for some money-making scheme (crypto or whatever) OR you're connected to people in power OR you personally piss off someone who is a hacker -- it is very unlikely you'll get screwed over due to those vulnerabilities :P
Just a point to add: this backdoor was (likely) planned years in advance; it took ONE guy a couple weeks (after the malicious code was released) to find it because he had nothing else going on that evening.
I'm relatively confident that the FOSS community has enough of that type of person that if there are more incidents like this one, there's a decent chance it'll be found quickly, especially now that this has happened and gotten so much attention.
Even better: there are also backdoors in closed-source software that will never be found. There may be fewer backdoors inserted, but the ones that get in there are far more likely to stay undiscovered.
You don't. Hackers often exploit things like this for ages before they are found. Every bit of non simple software also has bugs.
But chances are you won't be the target.
Just keep everything updated and you should be alright.
And implement least privilege and automatic updates
That's the neat part, we don't!
...but, we at least can have a shot of finding them.
In the meantime, I'd advise you to keep an eye out and maybe look into threat models. As people said in this thread already, bad actors probably don't care about your personal photos.
Unless OP is a celebrity or politician. Or knows they have an enemy with the resources to find and exploit potential backdoors.
I would say you can't, but if you are using open source software, then somebody can and will find them eventually and they will be patched. Unlike with closed source software, you will never know if it has a backdoor or not. This whole episode shows both the problems with open source, being lack of funding for security audits, and the beauty of open source, being that eventually it will be detected and removed.
We don’t.
Permanently Deleted
I'm pretty sure most closed source software is already backdoored.
It's a feature!
Afaik, most phones are backdoored that can be abused using tools like "pegasus" which led to a huge indignation in Hungary. I don't belive PCs are exceptions. Intel ME is a proprietary software inside the CPU, often considered as a backdoor in Intel. AMD isn't an exception. It's even weirder that Intel produces chips with ME disabled for governments only.
They are not produced for governments only, almost every consumer-grade CPU can have its ME disabled or at least scuttled, thanks to efforts like me_cleaner!
Have you tried it?
We can't know. If we would know, those weren't undetected backdoors at all. It's not possible to know something you don't know. So the question in itself is a paradox. :D The question is not if there are backdoors, but if the most critical software is infected? At least what I ask myself.
Do you backups man, do not install too many stuff, do not trust everyone, use multiple mail accounts and passwords and 2 factor authentication. We can only try to minimize the effects of when something horrible happens. Maybe support the projects you like, so that more people can help and have more eyes on it. Governments and corporations with money could do that as well, if they care.
The main solace you can take is how quickly xz was caught: there is a lot of diverse scrutiny on it.
Hmm, not really. It's only because it nerd-sniped someone who was trying to do something completely unrelated that this came to light. If that person has been less dedicated or less skilled we'd still probably be in the dark.
Check the source or pay someone to do it.
If you're using closed source software, its best to assume it has backdoors and there's no way to check.
Even if there are nation state level backdoors, your personal server is not a valuable enough target to risk exposing them. Just use common sense, unattended-upgrades, and don't worry too much about it.
I do IT security for a living. It is quite complicated but not unrealistic for you to DIY.
Do a risk assessment first off - how important is your data to you and a hostile someone else? Outputs from the risk assessment might be fixing up backups first. Think about which data might be attractive to someone else and what you do not want to lose. Your photos are probably irreplaceable and your password spreadsheet should probably be a Keepass database. This is personal stuff, work out what is important.
After you've thought about what is important, then you start to look at technologies.
Decide how you need to access your data, when off site. I'll give you a clue: VPN always until you feel proficient to expose your services directly on the internet. IPSEC or OpenVPN or whatevs.
After sorting all that out, why not look into monitoring?
Fun fact, you can use let's encrypt certs on a internal environment. All you need is a domain.
I do use it quite a lot. The pfSense package for ACME can run scripts, which might use scp. Modern Windows boxes can run OpenSSH daemons and obviously, all Unix boxes can too. They all have systems like Task Scheduler or cron to pick up the certs and deploy them.
We don't. That's why we use multiple layers of security. For example keeping all services accessible only via VPN and using a major OS that a lot of production workloads depend on such as Debian, Ubuntu LTS or any of the RHEL copycats. This is a huge plus of the free tier of Ubuntu Pro BTW. It's commercial level security support for $0. Using any of these OSes means that the time between a vulnerability being discovered, patched and deployed is as short as possible. Of course you have to have automatic security updates turned on,
unattended-upgradesin Debian-speak.you don't and will never will. I would recommend reading a lecture by Ken Thompson the co-creator of Unix for more details on this https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf
We dont.
if you are self hosting and enjoy over-engineering systems... VLANS, ACLs between subnets and IDS/IPS should be part of.your thinking. separate things into zones of vulnerability / least-privilege and maintain that separation with an iron fist. this is a great rabbit hole to fall down if you have the time. however, given a skilled adversary with enough time and money, any network can be infiltrated eventually. the idea is to try to minimize the exposure when it happens.
if the above is not a part of your daily thinking, then don't worry about it too much. use a production OS like Debian stable, don't expose ports to the public internet and only allow systems that should initiate communication to the internet to actually do so (preferably only on their well known protocol ports - if possible).
We don't know. But if there were well known backdoors to mainstream security practices we might see more companies that depend on security shutting down, or at least shutting down their online activities. Banks, stock trading, crypto exchanges, other enterprises that handle money, where hacking would be lucrative.
I don't think you need to worry about backdoors with most of those. Worry more about unfixed security holes due to an extreme emphasis on "stability" as in using old versions when fixes have already been released when it comes to anything hosted by large companies.
There's a concept of acceptable levels of risk. Companies are not going to shut down out of fear, or miss out on the business opportunities of online presence. There's money to be made.
Even with things as serious as spectre allowing full dumping of CPU and RAM contents simply by loading a website, I can't think of a single company that just said "well shit, better just die".
Serious, potentially business ending, security issues usually have a huge amount of effort when discovered put into mitigations and fixes. Mitigations are usually enough in the immediate "oh shit" phase. Defense in depth is standard practice.
There are several known instaces of crypto exchages getting hacked.
This was likely a state-sponsored attack. If your SSH isn't exposed to the internet this probably wouldn't have effected you. Also most people run stable distros like Debian on their server, and this particular vulnerability never made it to the stable branch. I would guess that most of the computers you have ever used have backdoors. Even if you run Linux (which may itself have some) you might still have a proprietary UEFI on your motherboard. Something like xz wouldn't effect you because no government really cares what's on your server. Smaller attacks can be avoided through common sense. Some people will expose services that require zero authentication to the internet. Follow basic best practices and you will probably be fine.
Edit: Also remember, google photos once flagged a picture of a child that the child's father had taken for medical reasons as abuse, so self-hosted may not be completely private or secure, but it's better than the alternative.
We don’t know. However, no one cares about your personal photos ; no one will ever attempt to hack you specifically unless you’re a high value target (in which case, stop hosting your photos anywhere immediately)
The only thing that could get your photos is if an undiscovered backdoor is exploited by someone doing some sort of a mass attack. As far as I know, they’re pretty rare, because people with the means to do them generally have a specific set of people they care about (which you are unlikely to be a part of).
We don’t know. However, no one cares about your personal photos ; no one will ever attempt to hack you specifically unless you’re a high value target (in which case, stop hosting your photos anywhere immediately)
To those assumptions I would say : we don't know. Personal vendettas do exist and we cannot look into the minds of individuals going crazy neither.
That fair enough I guess, really depends on what kinda personal photos you have. I know people are worried about revenge porn, I personally think the only actual remedy is not having any porn of yourself anywhere unless it’s your job 🤷🏻♀️
no one will ever attempt to hack you
My brother in Christ, how do you think botnets get built?
They did say specifically. I think bothers are usually automated attacks.
unfortunately, mass attacks happen all the time. if you ever had access to the authentication logs of an SSH server with an unfirewalled port 22 into the internet that has been running for a few months, you would see international IPs starting port scans and brute force attacks. there is always someone out there trying to hack random IPs. it's fucking wild west out there.
Yeah. I'd recommend using ssh keys and disabling password authentication whenever something is exposed to a public network
My ssh auth logs show a lot of login attempts from chinese IPs. That prompted me to install Fail2Ban..
There probably are, there's a reason why super high security systems aim for airgapping of sorts, and even that's not immune.
Good question. I have asking myself the same thing as well. In case of ssh it is possible to use 2FA with a security key, which is something I'd like to put in my todo.txt
The best you can do is use OSS software that has been battle tested. Stuff like OpenSSH and OpenVPN are very unlikely to have backdoors or major vulnerabilities currently being exploited. If you don't trust something to not be vulnerable, you're best to put it behind a more robust layer of authentication and access it only by those means.
I'm not a security specialist either. I learn new things every day, but this is why my NextCloud is accessible through TailScale only and I have zero ports exposed to the outside world.
The only real convenience I lose is being able to say "check out this thing on my personal server" with a link to someone outside my network, but that's easily worked around.
Next: how do we know tailscale's network hasn't been backdoored?
Headscale. And then you don't even have to trust any outside auth provider to not log in in your name.
I figure there's a certain amount of trust you have to have in strangers for a LOT of things we use every day.
I try to be selective with where I put that trust, especially when I can't just homebrew an advanced custom solution, but I figure Tailscale is much better than attempting to just host it on my LAN with an open a port to the big scary web and hope a bot doesn't find a gap and ransomware it all lol.
3-2-1 backups and a certain bit of trust.
Because heck, even CPUs have been found with exploitable microcode. (Spectre and Meltdown?) At some point you just gotta balance "best rational protection" with not going insane, right?
Headscale mentioned here is pretty neat too, but I feel like spinning up Dockers on Proxmox and Tailscale is as much moving parts as I'm willing to manage alongside everything else in life. :)
I think you can use Tailscale Funnels for that.
If backdoors exist, they’re probably enough to get your data no matter where it’s stored, so self hosting should be fine. Just keep it up to date and set up regular automatic backups.
Ah shit we are back to "Ken Thompson Compiler Hack" again
for those unfamiliar: Reflections on trusting trust by Ken Thompson
You can't trust any of it to be totally secure, it's effectively impossible. But, this is true of all software, at least open source is being audited and scrutinised all the time (as demonstrated).
All you can do is follow best practices.
You never know what security holes exist until they're exploited. Nothing you can really do about that. Security and convenience have always, and will always, be a trade off and a matter of personal acceptability. If you host anything, it will be potentially vulnerable, way less so if you take proper precautions. If you're not just overtly insecure, you'll probably be fine, but there's no way to say for sure.
Permanently Deleted
That's the neat part, you don't. However if you stay up to date it is not a big deal.
Well not too up to date as we just have witnessed 😁
Just use a stable system
While others are saying “no one cares about your personal photos” personal info is not the target of backdoor attacks like this. It’s more likely an attempt to get access to lots of processing power for a crypto mine or botnet.
It’s best practice to have the minimum packages required to run whatever service you are running, don’t add other stuff that you won’t be using. Using a distro that is “outdated” like Debian stable can help since the packages have had more time out in the wild to be tested.
I am sure that the xz incident has raised a lot of alarms across many projects.
The alternative is you could use a Windows server... Bear in mind a lot of code for windows is private and isn't open to scrutiny from the public so chances are there could possibly be built-in backdoors no one is aware of except for Microsoft or there are backdoors which are only known to a hacker who's discovered them.
The recent xz backdoor, I think is a stellar example of how powerful open source software can be, a community came together and noticed this in such a short period of time and had a way to quickly resolve this without this being a large scale issue, looking back at things like Log4j which caused so much pain across the entire industry, we are so lucky to have open source software communities.
@mfat It's the old problem about bugs. To know that a piece of software has no bugs you should be able to count them and if you could do it then should be able to locate them and make a fix. But you can't then there's no way to know there's no more undetected backdoor
Of course being open source helps a lot but there's no solver bullet
As many have pointed out, you don't know that there is not a back door in your software.
One way to defend against such an unknown is to have a method of quickly reinstalling your system, so if you ever suspect you have been compromised you can reload your OS from scratch and reconfigure it with minimal fuss. This is one reason I recommend folks learn one of the configuration management systems like ansible or puppet, and use those to configure your Linux servers. Having config management also helps you recover anfter unexpected hardware failures.
Defense is done in layers. No one layer will protect you 100%. Build up several layers that you trust and understand.
Very good points. Call me paranoid but i always fear I might fail to notice the symptoms of something nasty going on. I wish linux had a built-in, easy-to-use auditing/alerting system. I know this can be achived by experts but others have no idea what's actually going on on their machines.
Dude, you're so not paranoid. This stuff has happened to me. I had a Wordpress blog that was hacked and the exploit was stored in the DB so even after reloading the OS I still was infected because I hadn't sanitized my database. Luckily it was just Google search viagra spam, and it was a valuable lesson.
We don't
@mfat It's the old problem about bugs. To know that a piece of software has no bugs you should be able to count them and if you could do it then should be able to locate them and make a fix. But you can't then there's no way to know there's no more undetected backdoor
Of course being open source helps a lot but there's no solver bullet
How do you know what you don't know?
That's basically what you're asking. If you have an answer to that general question, it will answer your specific question.
Anti Commercial AI thingy
