Yeap you are right, custom firmware is the wrong phrase perhaps my fastidious friend :P. I suppose you can call it "cloud replacement" software that runs on the robot after you flash a rooted firmware that allows you to make changes.
You can check out Valetudo. It is a custom firmware you can flash on many robot vacuum cleaners. Think of it as a layer between the robot and the manufacturer's cloud that exposes the robot's capabilities locally thus making the connection with manufacturer's cloud redundant. There is a section with supported robots and instructions on their site making it very easy to install.