Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN.
Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
I don't understand why people want to take down websites. Especially sites like Lemmy, which isn't exactly sticking it to anyone because no one owns it!
In case you haven't considered this, some helpful advice.
To keep them from the lemmy.world door after the CDN installation
Change the public IP addresses
rotate your certificates
block all traffic appart from the CDN and only allow a limited known good IP addresses (like yours and your support team).
These steps will make your server harder to find, hopefully they move on.
Most of these ‘attacks’ are targeted at the database
A major PostgreSQL performance issue, logic mistake, was discovered today in lemmy_server and is an easy fix. Details: https://lemmy.world/post/2008987
Growing pains. This server and the platform will be better for it. If not for these script kids, some other attacker would eventually be motivated to try it.
Thank you as always for the transparency. This instance is going to be the most targeted because of its size.
Y’all dealing with this is hard but you’re going to figure things out that will help the other instances.
It's not. People hate large companies that have a dominant position in their industry. Usually, that's fair. However, in the case of DDoS protection, you have to have a large overbearing presence to be able to have the capacity to withstand such attacks. People don't know how to see through what's typically true for what's true in this case. Do I like having a dominant player in an industry? Not particularly. Do I understand why it's necessary in this case? Yes.
Come on everyone, let's be better than this. Ruud literally said script kids, why do yall have to go and blame reddit? The Lemmy gets more attention, and chaotic dumbasses do their thing. You don't have to do any mental gymnastics to tie it back to spez.
Cloudflare isn’t bad per se, but having huge amounts of the public internet behind a centralized provider is bad for the flexibility and resiliency of the internet as a whole.
On the plus side watching you all tackle and solve these problems gives me confidence in the long term viability of Lemmy and the fediverse. The transparency and often detailed technical discussion definitely helps a lot too.
Also, ping is now from 200-300 miliseconds to just between 50 and 60 (depending on your ISP):
64 bytes from 172.67.218.212: icmp_seq=1 ttl=64 time=56.2 ms
64 bytes from 172.67.218.212: icmp_seq=2 ttl=64 time=60.2 ms
64 bytes from 172.67.218.212: icmp_seq=3 ttl=64 time=55.8 ms
64 bytes from 172.67.218.212: icmp_seq=4 ttl=64 time=58.9 ms
64 bytes from 172.67.218.212: icmp_seq=5 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=6 ttl=64 time=60.5 ms
64 bytes from 172.67.218.212: icmp_seq=7 ttl=64 time=60.1 ms
64 bytes from 172.67.218.212: icmp_seq=8 ttl=64 time=55.0 ms
64 bytes from 172.67.218.212: icmp_seq=9 ttl=64 time=60.0 ms
64 bytes from 172.67.218.212: icmp_seq=10 ttl=64 time=61.4 ms
64 bytes from 172.67.218.212: icmp_seq=11 ttl=64 time=59.3 ms
64 bytes from 172.67.218.212: icmp_seq=12 ttl=64 time=58.5 ms
64 bytes from 172.67.218.212: icmp_seq=13 ttl=64 time=56.0 ms
64 bytes from 172.67.218.212: icmp_seq=14 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=15 ttl=64 time=58.7 ms
Damn these script kiddies.. I don't like Cloudflare at all but it does its job well. It may just be my paranoia, but putting a single entity in control of so many websites seems dangerous. I think we have all learned about the intentions of big corporations. But hey, it's better than being taken down tbf.
I put this site behind cloudflare in response to this post. Other than having to change SSL/TLS encryption mode to Full, it seemed easy. I turned on bot fight mode and I'm using the managed WAF ruleset that comes with the free tier. Any configuration recommendations anywhere in the panel?
I wonder now with the semi-adversarial/semi-cooperative nature between lemmy instances, if wer'e not going to see more DDOS and other types of raids happening because a different instance has an ax to grind against yours. Say between you defederated them, or they consider your instance too big etc.
Is it just me, or is old.lemmy.world still using its old IP address?
lemmy.world and m.lemmy.world
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=1 ttl=56 time=46.9 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=2 ttl=56 time=50.3 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=3 ttl=56 time=48.0 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=4 ttl=56 time=50.1 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=5 ttl=56 time=50.1 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=6 ttl=56 time=50.2 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=7 ttl=56 time=47.0 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=8 ttl=56 time=54.0 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=9 ttl=56 time=50.1 ms
64 bytes from 104.21.53.208 (104.21.53.208): icmp_seq=10 ttl=56 time=49.8 ms
--- lemmy.world ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9010ms
rtt min/avg/max/mdev = 46.893/49.635/54.048/1.956 ms
Not sure if it’s related, but today on Mastodon, I’m unable to upload photos. Also can’t see pics from other users. Profile pics are mostly greyed out too.
Script Kiddies are definetely some of the saddest people on the internet. If you're gonna be an unethical hacker at all, actually do it. Don't be a sissy.
Yeah, this is just growing pains for any website. Get popular enough for it to be "fun" to target. Then get enough data that it's "profitable" to target. Etc. And the usual way to deal is to first use an external solution at least until it becomes too expensive due to traffic volume. Then make your own solutions for problems you can solve yourself and pay external companies for the ones you can't.
How does cloudflare work? Do you install the private SSL certificate there and so cloudflare can see all traffic, including passwords, in plain text or is the path from browser through to your server still encrypted?
I had a hard time signing in the other day as I got confused in the instances but otherwise I'm enjoying the experience browsing here using the summit app.
Well I signed today and I got an error saying rate limit earlier for using these types of symbols "î¦âö)ééäë((ºÜݨ¿ã¿ï" I'm assuming It has nothing to do with this but just In case I'm making a comment about it edit:also just realized It may have been from how long the password was (33 characters)
Cloudflare isn't bad per se, but having huge amounts of the public internet behind a centralized provider is bad for the flexibility and resiliency of the internet as a whole.
http://crimeflare.eu.org list reasons why not to use Cloudflare, though IDK if it's just ultra-privacy oriented warnings or something else...
Not sure if I should be upset, although the claim of CF potentially sniffing passwords/credit card details/other sensitive information across various websites sounds plausible to me (some websites even have a TLS cert verified by "Cloudflare, Inc."!) 🤷♂️
Why don't you close the subscription to Lemmy.world so new people will subscribe to smaller server so that if one has problem, not the majority of the people are affected by it?
Isn't this supposed to be one of the main characteristic of the fediverse?
Am I the only one who'd have no issue with ads in lemmy? As long as they dont use too much space. Amount in ads in RiF was good for me, don't know if they earned much from it.
Money could go to app creator and instance owner, i don't care, as long as it helps running the community.
We was behind Cloudflare since day one 😀
And even on Cloudflare there is not the origin IP its again reverse proxied, and we are small site compared to lemmy.world 😜
Wouldn't be surprised if Cloudflare itself was hiring out blackhats to DDoS attack certain websites in order to get them into the fold, like racketeering. I mean this is America, I wouldn't put it past any company here, even ones pretending to be "virtuous"