Boston Dynamic's robots are works of art - the pinnacle of engineering - but its all designed movement. By this I mean the control systems, their movement plans - it is built and designed by experts in their field. It's not quite as simple as "go from A to B and do some parkour on the way". There's a very large gap between "what is mechanically possible to do" and "Just let the robot figure out how to do that".
Mechanically we're ahead of software for manipulation and kinodynamic planning.
I'm actually working on this problem right now for my master's capstone project. I'm almost done with it; I can have it generating a series of steps to try and fetch me something based on simple objectives like "I'm thirsty", and then in simulation fetching me a drink or looking through rooms that might have a fix, like contextually knowing the kitchen is a great spot to check.
There's also a lot of research into using the latest advancements in reasoning and contextual awareness via LLMs to work towards better more complicated embodied AI. I wrote a blog post about a lot of the big advancements here.
Outside of this I've also worked at various robotics startups for the past five years, though primarily in writing data pipelines and control systems for fleets of them. So with that experience in mind, I'd say we are many years out from this being in a reasonable product, but maybe not ten years away. Maybe.
tldr I write about some of the more interesting works that shaped my understanding of applying LLMs for AI agents and robotic applications. Introduction What is this LLMs as a fad - a caveat Are LLMs actually going to be useful for robotics? Instruct based Benchmarking LLM basics n-shot and rea...
I was aiming to use LLMs with robotics in an upcoming project, and needed to first verse myself in what is the current must-know techniques in the space. To that end I read a ton of papers and wrote this article to try and suss out the best parts of current state of the art.
I hope this helps people; I'd be thrilled to discuss much of this as well!
tldr I write about some of the more interesting works that shaped my understanding of applying LLMs for AI agents and robotic applications. Introduction What is this LLMs as a fad - a caveat Are LLMs actually going to be useful for robotics? Instruct based Benchmarking LLM basics n-shot and rea...
I was aiming to use LLMs with robotics in an upcoming project, and needed to first verse myself in what is the current must-know techniques in the space. To that end I read a ton of papers and wrote this article to try and suss out the best parts of current state of the art.
I hope this helps people; I'd be thrilled to discuss much of this as well!
An autonomous system is described that combines deep reinforcement learning with onboard sensors collecting data from the physical world, enabling it to fly faster than human world champion drone pilots around a race track.
A cool application of RLHF (Reinforcement Learning w/ Human Feedback - the same approach as what OpenAI used to train ChatGPT).
The authors trained an agent to fly FPV drones at a level surpassing world champions.
Towards Sample Efficient Robot Manipulation with Semantic Augmentations and Action Chunking
We use ChatGPT to allow a user to intuitively control multiple robots such as drones, robot arms, home assistant robots using only language.
I have argued for awhile now that the probabilistic nature of LLMs can represent a form of context that, when applied, can be utilized for robotic applications.
Seems I'm not the only one that had this idea. While simple, Microsoft researched applied a high level control library to demonstrate LLMs (ChatGPT) developing robotic task code.
ROS maintainers discuss popular robotics control and navigation algorithms in use within ROS2. The associated discussion can be found here.
If you're looking for what to study or try applying in your own projects next, this is worth a look.
This is the routine thread where we discuss what you'll be working on this week! A cool robot? A computer vision project? Something cool in reinforcement learning? 3d printing a drive train? Let us know!
Maybe instead you're studying something, or reading a paper that just came out? Post about it!
It’s also okay to say “nothing” too - it’s great for your mental health to take a break!
Looking for help? Ask a question! See someone working on something cool? Ask them about it! No project is too small or too "newbish"!
How're you liking the ID4? That and the ioniq5 are looking pretty good ATM... Though I wish the Honda E or ID3 was sold in the US...
Playing multiple co-op campaigns, and every druid across multiple campaigns experience, randomly, unexpected deaths when being knocked out of wild shape. Not death saving throws either - straight into full blown d-e-a-d.
Per the rules, it should go from wild shape -> spill over damage being removed from the remaining HP of the druid. Reading the combat logs we see that while enough damage is done to knock the druid out of their wild shape, the remainder is not nearly enough to put them into death saving throws, let alone outright kill them.
Is anyone else encountering this issue?
One addition- sunlight is very bad for resin printers, so if your garage has windows or you routinely keep your garage doors open keep this in mind.
tldr I created a custom robotic arm environment where the arm is rewarded if it sorts shapes a certain way, and then punished if it knocks them off a table. I used OpenAI’s PPO method to be somewhat successful. If you want to skip to the code, go here. The why Reinforcement learning is a topic that’...
In my first attempt at a long form technical post, I talk about a project where I had to use deep reinforcement learning to try and solve a robotics application. It worked ok - the post talks about my struggles, solutions, and what I'd change up in the future.