2d ago

How can I view logs from last session?

I have a server running Debian 12. It seems to hang up and freeze sometime between 12 hours and like 2-3 days. I really want to see what's happening, but I can't access my containers with the web UIs, can SSH, and adding a monitor shows the DE completely frozen. Rebooting fixes it for a while, but it locks up again after some time.

I had the same issue with 10 year old hardware so I threw together a "new" machine with some spare parts and it still happens I also tried rocky Linux and the same thing happened on both machines.

Any help and direction would be greatly appreciated!

12 comments

This app is quite useful for journald logs.
https://apps.kde.org/en-gb/kjournaldbrowser/

You can use journalctl -b <index>, where 0 is the current boot session, -1 the previous boot session and so on.
You can see all sessions with journalctl --list-boots if you want to pick a specific one.

One of the part that gave me random reboots was the PSU. It used to drop voltage randomly, affecting random components. After changing that it was rock solid.
If you carry the same PSU from ur older server maybe try changing it.

A few decades ago I bought a used IBM as a *nix server, but it would lock up at nearly random intervals like you describe. Tried a different Linux distro... same issues. Tried BSD - same issues!
It wasn't until after I learned of the 1999-2007 capacitor plague that I inspected the motherboard and saw that yes, several of the capacitories were bulging.
https://www.robotroom.com/Faulty-Capacitors-1.html
I mailed the motherboard to a servicer who replaced all the capacitors for a nominal fee. After that it was a rock solid system. You mention that this is recent hardware, but I would still suggest taking a peek at those caps.
- That's a fair point too. I'm really hoping it's not a hardware problem, but I'll definitely take a look at that

Someone already linked to journalctl, but if you just quickly want to look, the command journalctl and the flag —since
<your time here>
will get you going.
Journalctls output can be piped, so if you know what you’re looking for you can grep it easily.
- Yeah I really don't know that I'm looking for unfortunately. I was something would jump out at me. I'm pretty new to Linux altogether so I'm kind of winging a lot
  
  So go ahead and take a look at your journalctl output. The left hand side should be timestamps, so you can immediately figure out if it’s starting a million years in the past or sometime you know you had the problem.
  If it is a million years in the past, use the —since flag and specify the time you want to start at as enumerated in the manual file (man journalctl).
  Once you’re looking at the logs in journalctl from a day you know the problem happened, go ahead and use arrow keys and pgup/pgdn to find a reboot. You’ll know when you find a reboot because it’ll look different. The messages will be about figuring out what hardware is attached and changing runlevels and whatnot.
  Once you found where the reboot is, go backwards to find something weird happening in the logs.
  E: By default the parser (program used to handle text) of journalctls output is “less”. If you want to get out of it, press “q”, and if you want to know more “man less”.

https://www.loggly.com/ultimate-guide/using-journalctl/
- Thanks, I'll see what I can find with that
  
  also: if you don't find a smoking gun, you can try increasing logging levels on your stack to get more info.

12 comments