Skip Navigation

Is there a way to run old bare metal hardware on LAN for a dedicated computing task like AI?

This is an abstract curiosity. Let's say I want to use an old laptop to run a LLM AI. I assume I would still need pytorch, transformers, etc. What is the absolute minimum system configuration required to avoid overhead such as schedulers, kernel threads, virtual memory, etc. Are there options to expose the bare metal and use a networked machine to manage overhead? Maybe a way to connect the extra machine as if it is an extra CPU socket or NUMA module? Basically, I want to turn an entire system into a dedicated AI compute module.

8

You're viewing a single thread.

8 comments
  • Stop. Go back. This is the wrong way.

    If you're running python you basically need a full os.

    There are projects that run as an rtos, and in fact I worked on an ml soc that ran Linux, but there are 2 levels here:

    1. The ml processing itself, ie the math. This is simple in software and very complex otherwise. The software just says "copy this block and start running a matrix multiply". The hard logic is in moving data around efficiently.

    2. The stack. This is high level, python or so, and has graph processing overhead too. This needs a lot of "overhead" by its nature.

    In either case, don't worry about any of this, the overhead won't be very noticeable, you'll be cpu gated hard, the main thing is finding an optimized pytorch library.

    If you have an amd cpu or somehow have an nvidia gpu in your laptop you might be able to use their pytorch library which would improve performance by roughly 1.5-2 orders of magnitude.

    Unfortunately there isn't a pytorch implementation for Intel igpus, but there is an opencl backend for pytorch, and apparently this madlad got it working through opencl on an Intel igpu: https://dev-discuss.pytorch.org/t/implementing-opencl-backend-for-pytorch/283/9

    But don't worry about overhead, it's less than fractions of percents in these kinds of tasks and there are ways to bypass them completely.

    • Seems like avoiding context switching and all the overhead associated would make a big difference when pretty much everything in cache is critical data.

      I'm more curious about running something like Bark TTS where the delay is not relevant, but it would be cool to have the instructional clarity of the Halo Master Chief voice read me technical documentation, or test the effects of training my own voice, tuned to how I hear it, reading me stuff I find challenging. If the software is only able to process around 10 seconds at a time, just script it and let it run. The old machine will just collect dust otherwise.

      Anyways, what's the best scheduler with affinity/isolation/pinning?

You've viewed 8 comments.