[Troubleshooting] Random EOF shutdowns from Klipper.
Hi everyone, a week ago my printer (heavily modified Neptune 3) started randomly shutting down in the middle of prints. I come back to a print with the "Klipper reports: SHUTDOWN / Lost communication with MCU 'mcu'" error message.
The printer has been "under construction" for the last couple of weeks, but it has been in varying states of "working" for most of the time - working well enough for me to print the parts I needed to get it back to "fully operational". During this time, the printer never shut down like it is now.
Only once I started making little cosmetic changes did the problem present itself. I was running a known-good print, and I got the above error twice (first time after ~2 hours, second time after ~1 hour) before I got a successful print off of it. This was last week.
After this successful print, I continued other prints with no issues. After a day or two with no problems, an hour long print threw the error at me four consecutive times between 10-45 minutes into the print. This is when I started looking into my klippy log and found some relevant articles citing things like EMF interference, bad power supplies, faulty cables etc. I realized that one of the changes I had made rerouted the printer USB cable right around the Z-stepper, so I rerouted it to how it was originally and immediately managed a successful print. This was 5 days ago.
After moving that cable I had no issues with printing several-hour long prints... until last night. I had been printing all day, then the problem came back. After one print finished, I queued up another print with a plate full of parts, it failed after 1.5 hours. Tried the same print again, failed in 30 minutes. I re-sliced to only a handful of parts to see if I could get those to print before the error occurs, and it's failing 15 minutes into the print.
The printer power supply is the unit that came with the Neptune, and it isn't powering anything besides stock hardware (exception being the SKR mini board), so I don't think it's that. The pi is on a quality unit. The USB cable has been working for a long time so I also don't suspect that, but I'm probably going to buy a new one today just to be sure. I adjusted my enclosure setup so that the Pi and SKR are able to get cool air (at one point had a personal fan pointing at the open electronics box, still failed).
Here is a link to my most recent klippy log (abridged to the start of the last failed print). I'm not very familiar with reading through this and finding oddities, but I do think it's strange that it seemed to load my preheat script in the middle of printing right before the EOF error. (It should be noted that this preheat script was made 1 or 2 failed prints before this most recent one, so it isn't the source of the error as prints were failing before the script was made). If there's anything I'm missing or something else I can try, please let me know!
Edit: While typing this post, I was running the same failed print without filament and both heaters turned off. It ran for about 45 minutes (most recent failure occurred at 12 minutes) so I cancelled the print and started it again with heaters turned on, still without filament. It again ran for about 45 minutes, so I again cancelled it and started the print again, this time with filament loaded. It failed in 5 minutes.
Edit 2: A test print with heaters on and no filament failed after 1h8m. So it isn't an issue with extruding filament.
Edit 3: New cable with the 5v leads taped off per @SzethFriendOfNimi@lemmy.world's advice. Ran the print without filament until completion. Reloaded the same file with filament, print ran without issue until the 1h14m mark, at which point I tapped my Klipperscreen device to wake up the screen, and as soon as it displayed the status, the printer errored out. This can't be a coincidence, can it? Whenever the print goes unmonitored for a long time, it fails as soon as I do something (load mainsail, turn on the klipperscreen) to check the status of it.
How are you powering the pi itself? Wondering if the wakeup draw from your screen is enough to make it unstable, 4 I believe having higher power drawn than the 3 if you're using the same power supply. Pi isn't oced either?
Quick Edit: I run my v2.4 on a lepotato with oodles of usb attachments (camera, multiple mcus, wifi and a screen. Replacing the mcu connections with a usb-canbus bridge when the heatwave ends) but with it connected directly to a meanwell 5v5a PSU into the gpio header, have never had communication issues to the octopus pro I use. Skr mini on the other printer is connected to a laptop host so it definitely has enough power, did have some odd issues with the skr mini and having an accelerometer connected to the spi header on boot and separately the 24v supply becoming loose that I fixed by crimping ferules onto the supply wires when I added a molex connector to make taking the printer out of its enclosure easier.
Pi is not OC'ed, but I did just realize that it's running on a power supply that came in a cheap starter kit and not the nicer one I'd ordered, so maybe that's an issue. You would think the Pi would report under voltage if that were the case though, no? When Klipper errors out, the pi doesn't shut off or report any issues.
I will say that I've completed two 4 hour long prints with all external USB devices disconnected, and when I tried a duplicate of one of the completed prints with the klipperscreen device (on a new nice cable) and my webcam (connected but not enabled via crowsnest) it failed after 3 hours.
You'd think right, anecdotally when I was meaaign with undervolting on my desktop, you'd get instability or crashes at the limits without useful errors. Could think maybe the processor isn't getting enough power and that's causing instability? Found a klipper thread that about losing connection with the mcu which totally can show up as an EOF error.
I suppose it could be a power issue based on that. That's the same article I linked in the OP; my klippy.log shows that it loses connection with the MCU, then a few lines later I get the EOF error.
Printer is finishing the third 4 hour print with klipperscreen device unplugged. Really leaning towards this route, I'll try swapping my nice power supply in when I get home and see if that helps.
Fingers crossed. There's hats you can get to power the pi off of printer's 24v supply, considering doing it on my klippered mk3s and switch that over to using a SBC instead of the laptop.
If you can swing it, definitely worth printing a spare set for the stealthburner + cw2, just in case you crack anything during assembly.
I found this instruction of assembly pretty entertaining. "Break it and try again without breaking it".
Actually have the SB already on the current printer. It was having extrusion issues for the better part of 3 months and I built it as a last ditch effort to save the printer from the closet. After building that I realized how much I loved the voron designs and once the printer started acting up again I made the decision to just start piece mailing a 2.4
Totally get you, I swapped my prusa's hotend for a stealthburner as well, they're super nice to service, especially with the 2 part PCB, but having the entire assembly mounted from the front is fantastic.
Stealthburner ended up fixing your clog issues entirely?
Stealthburner ended up fixing your clog issues entirely?
More or less. You can check my post history, made a lot of fuss about it here trying to get it squared away, in part it was caused by some magically reoccurring issue where my printer boards were reporting incorrect hotend temps (sometimes off by as much as 70°C). But even after fixing that side of things my old setup kept clogging; kept diagnosing back and forth between extruder and hotend, decided why not build a stealthburner and replace both, if that doesn't fix the problem then I'm part way done with building a voron.