Foundation Models for Humanoids

We have settled on this nice pre-training plus post-training recipe for getting a multi-modal foundation model to generalize (somewhat) to manipulation or loco-manipulation tasks.

However, this pipeline is finnicky. To me, this process of post-training is essentially overfitting. You are collecting 100-200 demonstrations (painstakingly) for your desired task. Then, you post-train and your model can perform your task... sometimes. In doing so, you lose any generalization from the pre-trained model. There is no other task it can effectively do now. Furthermore, it can only do what you demonstrated. If you move your humanoid a few steps back or a few steps to the right of where you collected your day, you're not going to get good performance.

With this work, we first develop a failure detection system for humanoids. This performs test-time reasoning with a VLM orchestrator agent plus a learned visual failure detector that run in tandem to determine if the foundation model is more or less set up to fail. In the event that it does fail, we also design a set of recovery tools including locomotion and manipulation behaviors that can be autonomously called to move the system back into an area of the environment where the model is predicted to succeed.

Want to learn more? Check out the following links:

• Paper