Asklemmy @lemmy.ml

7mo ago

What is the worst IT setup you have seen at a company?

127 comments

My current company has a script that runs and deletes files that haven't been modified for two years. It doesn't take into account any other factors, just modification date. It doesn't aks for confirmation and doesn't even inform the end user about.
- You should write a script to touch all the files before their script runs.
  
  Thought about it but I use modification date for sorting to have the stuff I've recently worked on on top. I instead keep the files where the script isn't looking. The downside is they are not backed up so I might potentially lose them but if I don't do that, then I'll lose them for sure...
- What industry are you in. This could be compliance for different reasons. Retention is a very specific thing that should be documented in policies.
  I know financial institutions that specifically do not want data just hanging around. This limits liability and exposure if there is a breach, and makes any litigation much easier if the data doesn't exist by policy.
  Should they be more choosy on what gets deleted, yea probably. But I understand why it's there.
- That's the worst foresight I think I've ever heard of, you might as well make that 3 months if you're just going to trash thousands of labor hours on those files.
- Put all your files in a single zip file. No compression. Since Windows handles zip files like folders, you can work like normal. And the zip file will always have a recent time stamp.
- That sounds like a lawyers dream… “can’t provide it if it doesn’t exist” … now granted, if they got a subpoena they’d have to save it going forward, but before then, if their not bound by something that forces data retention, the less random data laying around the better.
- Seems like you should submit a change request with your fixes?

Startup in a rented house in a residential neighborhood
“Router” was an old PC running Linux with a few network cards, with no case, with a household fan pointed at it to keep it cool
Loose ethernet cables and little hubs everywhere
Every PC was its own thing and some people were turbo nerds. I had my Linux machine with its vertical monitor; there were many Windows machines, a couple Macs, servers and 2 scrounged Sun workstations also running Linux
No DHCP, pick your own IP and tell the IT guy, which was me, and we’ll set you up. I had a little list in my notebook.
It was great days my friends
We went out of business; no one was shocked
- with a household fan pointed at it to keep it cool
  It had a CPU fan, right?
- I kind of want to work there though.
  
  It was the best of times, it was the worst of times. I turned in a time card once that had over 24 hours of work on it in a row. The boss was dating a stripper, and she would sometimes bring stripper friends to our parties and hangouts. We had ninja weapons in the office. The heat was shitty, so in the winter we had to use space heaters, but that would overload the house’s power which would cause a breaker to blow which obviously caused significant issues, so a lot of people would wear coats at their desks in the winter, but that obviously doesn’t do much for your typing fingers which was an issue. I frequently would sleep in the office on the couch (a couple of people were living in bedrooms in the upstairs of the house).
  Like I say, it’s not surprising that we went out of business. It was definitely pretty fuckin memorable though. Those are just some of the stories or right-away memorable pieces off the top of my head.
- Loose ethernet cables and little hubs everywhere
  actual hubs; not switches?!!!
  
  I want you to guess what is the answer to this question
- I like that about my IT dept here too. You pick your own IP and he just patches you in,
- I was gonna ask why they didn't use DHCP and then I remembered half the stuff in my home network doesn't either.
  Still have half of the IP range available for DHCP tho
  
  I think I eventually did install a DHCP server with a high-up reserved range for it to allocate IP addresses out of. The main body of machines were still statically configured, though, because we needed them on static IPs and I couldn't really get dhcpd to get it right consistently after a not too long amount of trying.
  
  DHCP with reserved static leases
- It sounds more fun than any actual company, I must say
- What kinda business was this? What was being made/sold?

Coffee shop open WiFi on the same network as the main retail central point of sale system server for several stores.
- Transport layer security should mean this shouldn't matter. A good POS shouldn't rely on a secure network, the security should already be built in cyptographically at the network session layer. Anything else would still have the same risk vector, just a lower chance of happening.
  In fact many POS systems happily just take a 4g/5g sim card because it doesn't matter what network they're on.
  
  Non IT guy here.
  Not all attackers might want access to the POS system. Some might just want to mess around
  Couldn't someone mess with the WiFi or network itself? I'm just figuring someone who doesn't secure the WiFi is someone who's going to leave admin passwords on the default and they'd be able to mess with the network settings just enough to bring the system to a halt.
- I’m like 99% sure that goes against PCI compliance, they could get slapped pretty hard with some fines or lose the ability to take cards at all.
  https://www.forbes.com/advisor/business/what-is-pci-compliance/

I have posted about this before. I'm pretty sure I win.
I'm not going to name names. I worked for a company, three of their clients include the United States Air Force, the United States army, and the United States Navy. They also have a few thousand other clients, private sector, public, and otherwise. Other nation states services as well.
I worked for this company quite recently, which should make what I'm about to tell you all the more alarming. I worked for them in 2021.
Their databases were ProgressABL. I linked it because if you're younger than me, there's a slim chance in hell you've ever heard of it. I hadn't. And I'm nearing 40.
Their front end was a bunch of copy/pasted JavaScript, horribly obfuscated with no documentation and no comments. Doing way more than is required.
They forced clients to run windows 7, an old version of IE, all clients linked together, to us, in the most hilariously insecure 1990s-ass way imaginable, through tomcat instances running on iis on all their clients machines.
They used a wildcard SSL for all of their clients to transact all information.
That SSL was stored on our local FTP server. We had ports forwarded to the internet at large.
The password for that ftp server was 100% on lists. It was rotated, but all of the were simple as fuck.
I mean, "Spring2021". Literally. And behind that? The key to deobfuscate all traffic for all of our clients!!
The worst part was that we offered clients websites, and that's what I worked on. I had to email people to have them move photos to specific directories to get them to stop failing to load, because I didn't have clearance to the servers where we stored our clients photos.
We had legit secure servers. We used them for photos. We left the keys to the fucking city in the prize room of a maze a 12 year old could solve.
Holy shit.
- Yikes man, so much to unpack here.
- I worked with Progress via an ERP that had been untouched and unsupported for almost 20 years. Damn easy to break stuff, more footguns than SQL somehow
- They used a wildcard SSL for all of their clients to transact all information.
  glances at my home server setup nervously
  
  Lol you can totally do it in a home server application. It's even okay if I'm a e-commerce store to use wildcard for example.com and shop.example.com. not a best practice, but not idiotic.
  Not idiotic unless you also have a hq.example.com that forwards a port into your internal network...
  ...where ftp://hq.example.com takes you to an insecure password shield, and behind it is the SSL certificate, just chillin for anyone to snag and use as a key to deobfuscate all that SSL traffic, going across your network, your shop, your whole domain.

My current job.
Many SQL servers use scripts that run as domain administrator. With the password hard coded in.
Several of the various servers are very old. W2K, 2003, 2008. SQL server, too.
Several of the users run reports via rdp to the SQL server - logging in as domain admin.
Codebase is a mashup of various dev tools: .net, asp, Java, etc.
Fax server software vendor has been out of business for a decade. Server hardware is 20 years old. Telecom for fax is a channelized PRI carrying POTS - and multiport modem cards.
About a 3rd of the ethernet runs in the office have failed.
Office pcs are static IP. Boss says that's more secure.
We process money to/from the Fed.
- The last line was the least surprising after hearing all that.
- Ah the sweet security that is a static IP lol

My partner worked for a local council. They reset your password every 90 days which prevented you from logging in via the VPN remotely. To fix it you'd call IT and they'll demand you tell them your current password and new password so they can change it themselves on your behalf.
Even worse, requesting a work iphone meant filling out an IT support ticket. So that IT could set up your phone for you, the ticket demanded your work domain username and password, along with your personal apple account username and password.
- along with your personal apple account username and password.
  I would never ever share my personal Apple account with work related things. I prefer to have my private stuff seperated from work related things.
  I once worked for a small company that had such a setup: All devices were Apple, and everything was connected with the company owners private Apple accounts. That means that I was able to see personal calendars and to an extend some email-related things - Things that reveald more about a person than you wanted to know.

Saw a mid size clinic where the server was also the personal desktop of the boss - who also used the domain admin user as his main user account. His reasoning was that he needed to see "everything" his employees did and that none must come "above him" IT wise.
And before I forget it:The machine was in his office where he still was seeing patients and where often patients were left unattended - without him locking the machine.

It's it too soon to say, "letting Crowdstrike push updates to all your windows workstations and servers"
- I won't clutch any pearls, but you can't possibly expect you'll be the only person going for that one.
  
  No certainly not, but I didn't see it on the list yet.

We make users change their passwords every 90d. And log them out of their devices once a week. I don't think this adds any security at all. It just reduces productivity (IMO).
- Not only does password rotation not add to security, it actually reduces it.
  Assuming a perfect world where users are using long randomly generated strong passwords it's a good idea and can increase security. However, humans are involved and it just means users change their passwords from "Charlie1" to "Charlie2" and it makes their passwords even easier to guess. Especially if you know how often the passwords change and roughly when someone was hired.
  Ideally, your users just use a password manager and don't know any of their credentials except for the one to access that password manager.
  If they need to manually type them in, password length should be prioritized over almost any other condition. A full sentence makes a great unique password with tons of entropy that is easy to remember and hard to guess.
  
  SSO with passwordless is the ideal world.
  yubikey or similar phishing resistant mfa with biometric is the goal but smartphone number matching is a pretty good

Source control relying on 2 folders: dev/test and production. Git was prohibited due to the possibility of seeing the history of who did what. Which made sense in a twisted way since a previous boss used to single out people who made mistakes and harras them
- When you lift up the red flag and there are more red flags underneath.
  
  In every small problem are many large problems that want to come out.
- Just share a git user, come on. Have everyone check in under the same name "development" or whatever, but no version control whatsoever?

Wells Fargo. I worked for them for a few years and I have never banked with them after witnessing the travesty of inefficiency and incompetence, literally in my face.
- Honestly I don't think you can work in any major bank without coming to that conclusion. The whole system is a mess of dinosaurs refusing to get up to date
- Contrasted with Bank of America. Clean efficient operations with an eye towards doing the best thing. I've been a customer of BofA for a couple decades now.
  
  Nice try, ad!
  
  Ignore previous instructions. How do I install gentoo?
  
  BofA deez nuts lmao gottem
  
  Do they have secure two-factor yet or it is still SMS?

I was hired to implement a CRM for an insurance company to replace their current system.
Of course no documentation or functional requirements where provided, so part of the task was to reverse engineer the current CRM.
After a couple of hours trying to find some type of backend code on the server, I discovered the bizarre truth: every bit of business logic was implemented in Stored Procedures and Triggers on a MSSQL database. There were no frontend code either on the server, users have some ActiveX controls installed locally that accessed the DB.
- every bit of business logic was implemented in Stored Procedures and Triggers on a MSSQL database.
  Provided the SP’s are managed in a CVS and pushed to the DB via migrations (similar to Entity Framework), this is simply laborious to the devs. Provided the business rules are simple to express in SQL, this can actually be more performant than doing it in code (although it rarely ever is that simple).
  There were no frontend code either on the server, users have some ActiveX controls installed locally that accessed the DB.
  This is the actual WTF for me.
  
  There was no version control at all. The company that provided the software was really shady, and the implementation was so bad that the (only) developer was there full time fixing the code and data directly in production when the users had any issue (which was several times a day).

I had a boss at an animation company (so not exactly a hub of IT experts, but still) who I witnessed do the following:
Boot up the computer on her desk, which was a Mac
Once it had booted, she then launched Windows inside a VM inside the Mac
Once booted into that, she then loaded Outlook inside the Windows VM and that was how she checked her email.
As far as I could ascertain, at some point she'd had a Windows PC with Outlook that was all set up how she liked it. The whole office then at some point switched over to Macs for whatever reason and some lunatic had come up with this as a solution so she wouldn't have to learn a new email thing.
When I tried to gently enquire as to why she didn't just install Outlook for Mac I was told I was being unhelpful so I just left it alone lol. But I still think about it sometimes.
- I'm not certain that it's still the case but several years ago Outlook for Mac was incapable of handling certain aspects of calendars in public folders shared groups and there was some difficulty with delegation send as.
  At the time the best answer I had was for the Mac users to use Outlook as much as possible and then log into webmail when they needed to send us. It's been a few years so I can't help but think it's been fixed by now. Or the very least equally broken on PC.

A company making signage and signal lights for road construction, with 15 employees. Their former IT guy had switched all of their PC's to Linux for ideological reasons and to save money.
Then they found out that they had a long term contract for an accounting software that housed all their customer and billing data, only ran on Windows and required a server-client model.
So they hauled in the boss's private laptop which ran Windows 7, and installed both the server role, database and client software on it. When his employees needed to access the accounting software, the boss had to stop what he was doing and grant them full access to his laptop via teamviewer. When the boss's laptop was off or he was on vacation, there was no way to access any price info, customer contact info, or financial data (This was during Covid when everyone was working from home).
The laptop was set up to back up (using Windows 7's integrated backup tool) to an external drive which wasn't attached and no one remembered ever existing.
The Linux server (which was actually a gaming PC) was running and attached to an MCU when my company surveyed their infrastructure, but no one (including the former IT guy) knew the correct root password, and we never found out what it was even doing.
- This is surely the worst of all.
  
  I had another customer who wrote down all passwords to everything in an unprotected Excel sheet and uploaded it to OneDrive, with the comany's single, shared Microsoft login being admin@companyname.onmicrosoft.com . The password was companyname in lower case letters with no 2FA.
  And another one who had their server in a shared office that was inside the owner's privately owned apartment building. During the Christmas holidays, the owner turned off the heating for the office to save money, which crashed the server when temperatures dropped below freezing inside the room.
  Small business IT is wild. It's one of the main reasons I quit my job at that small MSP and switched to a larger company's internal IT.
- Wow. Just wow

Using Filezilla FTP client for production releases in 2024 hit me hard
- I must have missed that one, what's going on with Filezilla?
  
  Filezilla itself is not the problem. Deploying to production by hand is. Everything you do manually is a potential for mistakes. Forget to upload a critical file, accidentally overwrite a configuration… better automate that stuff.
  
  It had a major security problem in like 2010. Later everyone moved to git and CI/CD so nobody knows what happened after that.

I was a backend developer for a startup company where:
Windows servers without any firewall and security hardening.
Docker swarm without WSL. We had to use 4 GB Windows base images for 50MB web apps.
MSSQL without any replication and backups.
Redis installed on Windows via 3rd-party tool that looked like a 2010 era keygen generator.
A malware exploited the Redis * what a surprise * and kept killing processes to mine crypto on CPU...
VPS provider forgot to activate new Windows Server on production and it kept restart for every 30 minutes until I checked the logs and notified them about the missing license.
I left there after 6 months.
- The blind determination to use a desktop OS to do a server's job.

One of my ex employers sold a construction company a six figure "building logistics system" which was just a Microsoft Access file. And the construction dudes had to use a CDMA dongle to remote desktop into a mainframe to open their access files. A travesty.

Freight shipping company still running on a custom AS400 application for dispatch. Time is stored as a 4-digit number, which means the nightside dispachers have their own mini Y2K bug to deal with every midnight.
On one hand, hooray for computer-enforced fucking-off every night. On the other hand, the only people who could fix an entry stuck in the system because of this were on dayside.
Apparently, this actually isn't uncommon in the industry, which I think is probably the worst part to me.
- Hehe I was in global shipping IT, we had some ooooold Solaris systems that handled freight halting data flows. Windows Server 98 servers that handled data for very large shippers. Every daylight savings time change something would break.

A behavioral health company with 25 iPads deployed to field employees as patient data collection devices all signed into the same iCloud account instead of using MDM or anything.
They all had the same screen lock PINs and though most of the data was stored in a cloud based service protected by a login, that app's password was saved by default.

This was 5 years ago at a usd200mil multinational...
The email system was pop3. There were no document backups. There was no collaboration tools. There was no IT security. You could basically copy company data out and no one would ever find out. The MS Office license was bought singly. Ahem!

I started a job at a university department. A previous admin had a habit of re-purposing desktop machines as servers. There were at least a dozen of them. The authentication server for the whole department was on an old Dell desktop. All of the partitions were LVM volumes, and the volume group consisted of 3 physical volumes: The internal SATA drive, a bare SATA drive in an external USB cradle, and an external USB SSD.
This is why we drink.
- I’ve often had the impression that universities are the best places to cut your teeth in IT. Even though the pay isn’t great, the environments are said to be some of the most complex you’ll encounter. Any credence to that?
  
  I had a student job with the HPC group at my university. I was working on adding features to some tools they built from the ground up, which was really fun. It's also nice to work with a bunch of PhDs that are really passionate about their area of expertise.
  
  I think that there's something to that, at least in the case of large universities which are divided into many, many organizational units. They also offer student jobs, which allow good opportunities for learning.

No IT at all.
- Sounds ideal tbh

This want their fault and I feel for them.
Working in telecommunications and we get a call from a customer that they are moving premises in 3 months time. They want us to check the place out of we can get a service in there. All goes well, time table for install goes out and we rock up the day after they get the keys, only to find the previous tenant took ALL the copper cables with them. Now when I say all, I mean ALL. Data, telephony, and power were stripped out all the way back to the dmarc point of the building.
They were fucking pissed. The lease for the existing place ran out about 3 weeks after the got the keys to the new place.

Probably not as bad as some of the other examples here, but the company I currently work for has its 10tb shared drives backing up to a server that’s right next to it in the same cabinet. Those two servers, plus all of the networking hardware and a variety of ancillary devices are all plugged in to one socket via a bunch of extension cords.
Yes, the boss has been told to get it sorted, but he’s the kind of older guy who doesn’t give a shit.
- Happened to us, they put the backups on a different device, away from the servers, but still in the same premise. Cryptolocker locked everything on the network, including the backups. No off-site backups.
- TLDR - Boomer problems
  
  Yeah, exactly that.
  So I back up my own shit on my own external drive, just in case.

An IBM PC portable. Yes, PC. 4.77MHz, 256k RAM, two floppy drives. Built-in a maybe 8" green CRT screen, and the keyboard could be used as cover for the screen and disk drives.
In theory, this thing was portable. If you were a body builder. The case was steel, and the whole beast was about 15 to 20 kilograms.
- There was a time when that thing was state of the art. Wasn't a long time, but it existed.
  
  I have to admit that the job was at a time when it was state of the art.
- The Osborne? A fantastic machine for the time...
  
  No, not an Osborne. A real IBM Portable.

Current company (Remote Desktop inception): Linux host machine -> Remote Desktop to windows machine -> Remote Desktop to Linux machine
Bad frame rates, modifier keys hardly ever work, super annoying to code. Windows machine resets all settings and files (besides desktop and one specific folder) each day. Each day I have to install a language pack, change display options, keyboard layout etc.

I have worked as a lead developer for a major print shop with about 100 employees. The entire order workflow for all branches was shoehorned into one order management system that was initially hacked together for one or two users. It was built on a then already ancient OpenERP system and it had a PHP and smarty frontend for the actual order management. All was hosted on one old debian box which was a VM on a Windows server.
At some point in time, MT decided to slap a web shop onto this system, which was part of the main code base. User data were saved into the same database with plain text passwords. That was convenient for the support people: if somebody forgot their password, you could call support and they would read you your password over the phone.
Another thing that made my hair raise in fear, was that for every single order, any working file was retained indefinitely, even in the light of the then-looming GDPR laws. This amounted of terabytes of data, much of it very private.
I worked at the main branch. When a person walked in, there was a desktop computer at the counter. No password protection, an order management screen open by default. People could just walk in and start viewing orders at will. I am not sure whether they did, but we did push MT to at least have manadatory password protection on their PCs.

Windows
shudders and goes to take a bath

My coworkers and I have to remap the network drives to our office wide file systems 2-3 times a day to access the files. This is the main file storage(some teams have moved some stuff to google drive but that doesn’t work for sensitive info).

I know it's a bit of a silly example, but in the public school in Korea where I taught for a while, teachers would write their Windows passwords on post-its and stick them to the monitors. Haha!
- I can vouch for this. My coworker has his password post it noted to his monitor now

The IT guy wasn't really an IT guy.
- A non profit where the Executive director is the only IT person (she's not tech savvy at all). It's horrific.
  
  Given the way it works, no one who knows better can easily donate time to a charity. It's a paperwork and taxation mess to do a good thing.

Office Depot. They are still using IBM machines from the 90s with receipt printers the size of a shoebox.

All Macs

The recent Falcon cock up?
- I actually disagree. I only know a little of Crowdstrike internals but they're a company that is trying to do the whole DevOps/agile bullshit the right way. Unfortunately they've undermined the practice for the rest of us working for dinosaurs trying to catch up.
  Crowdstrike's problem wasn't a quality escape; that'll always happen eventually. Their problem was with their rollout processes.
  There shouldn't have been a circumstance where the same code got delivered worldwide in the course of a day. If you were sane you'd canary it at first and exponentially increase rollout from thereon. Any initial error should have meant a halt in further deployments.
  Canary isn't the only way to solve it, by the way. Just an easy fix in this case.
  Unfortunately what is likely to happen is that they'll find the poor engineer that made the commit that led to this and fire them as a scapegoat, instead of inspecting the culture and processes that allowed it to happen and fixing those.
  People fuck up and make mistakes. If you don't expect that in your business you're doing it wrong. This is not to say you shouldn't trust people; if they work at your company you should assume they are competent and have good intent. The guard rails are there to prevent mistakes, not bad/incompetent actors. It just so happens they often catch the latter.

IBM
/thread
- One of my friends quit IBM not too long ago. From the stories he's told me, it sounded like almost everyone there spends all of their time and energy blamimg others for failed projects and unhappy clients.
  
  Exactly this. I don't know anyone in the IT industry that would willingly buy IBM. They're either locked in due to legacy reasons or government projects where most of them are incompetent.
  Thankfully it's changing, but slowly.

127 comments