I'm a researcher at Deepmind that contributed to these models. (And the opinions here are my own)
Just want to say, Deepmind is a great place to work and the only (Edit: one the few unique labs!) lab where you can move from large frontier models (Gemini), frontier open models (Gemma), robotics (what you see here), science (weather, biology, more) and basically any other topic related to intelligence. It's really an incredible place to be, with incredible people. Consider joining! And thank you for the enthusiasm here.
I’m a mechanical design engineer (10 years) turned software engineer (7 years in autonomous vehicles). Is there a particular role you would recommend for someone with my experience?
What happens when those ideas need to be scaled into real products, though? For instance, I can't really imagine Google being fully committed to manufacturing and selling robotic arms at scale.
Would you mind expanding on the difference between frontier model development and open model development? I was under the impression that they were nearly the same thing and would thus be handled by the same team. My impression is that open models tend to go through more safety testing and are generally smaller, but are otherwise the same thing. Shouldn't most of what is commonly referred to as 'AI research' be applicable to both?
I do think Allen Institution for AI (AI2) is has coverage across most of these domains ( albeit their frontier isn't nearly so far out, hopefully the $152m NSF awarded them + Nvidia is a fruitful partnership there).
Totally fair and thank you for nit picking! I edited my comment. AI2 is doing amazing stuff, I read their papers and code all the time and appreciate how open they are.
While Anthropic and Open AI get 80% of the attention here, it's impressive to see how much Google is doing: near frontier model, fast models, open weight models, image generation, video generation, music generation, robotics, etc.
It's certainly an odd situation where Google is on the forefront of all these emerging technologies while the latest chatbots from OpenAI and Anthropic are getting breathless press releases and wildly hyperbolic coverage ("rogue AIs" "too dangerous, must be banned" etc...)
Google acting like a normal but competent company that's just chugging away, meanwhile the hype cycle is propping up its upstart competitors to truly ludicrous valuations.
> Google acting like a normal but competent company that's just chugging away
Competent is a stretch. Google's AI offering seems to be, once again, PM led–lots of constantly-changing brands being merged and deprecated with zero customer support or service.
They'll almost certainly be one of the survivors. Their technical competece is unmatched. But that doesn't mean they have a great product in the way both OpenAI and Anthropic do. (Outside their datacentres, which are a legitimate feat.)
> Google acting like a normal but competent company that's just chugging away,
Chugging away not releasing products or releasing ones worse than their competition typically. Not sure that’s what a normal competent company should be doing.
Google wins on infrastructure and creativity (NotebookLM, etc) but entirely lacks product coherence. The ecosystem is a mess, just like Microsoft's offering, and I don't think this is a coincidence but rather a symptom of the type of company in which these two companies have evolved.
Furthermore, Gemini is just all around a less trustworthy and mature model, for many reasons. Very smart but lacking the precision and holistically exhibited in the more recent models from OpenAI and Anthropic. On the flip side, Google's work on Gemma is unmatched.
These robots look slow and not very fluid in their motions, but LLMs like ChatGPT also looked very dumb initially. If progress is as fast as LLMs , this could have massive applications in a few years.
For house keeping tasks the robotics companies love showing off... it honestly doesn't matter if it takes a robot longer than a human to clean your house. As long as it gets done before you get home from work, no big deal.
I'm bearish that they'll be economically viable in the house for a very long time.
For businesses the bar for adoption is very low: If the thing can work repetitive jobs for 24 hours a day and replace 3 shifts, the purchase bar is nominally anything less than 3 x human salary if your budgeting horizon is 1 year. That's a high number, and probably fairly easy to achieve.
For homes, it's a very different bar. You'd have a hard time convincing most American families to purchase anything with a >$1000 price tag. Currently that's pretty much impossible for a humanoid.
Exactly. People buy those automated circular floor vacuums. Anyone can get it done with a regular broom/vacuum in a small fraction of the time it takes for those things. Just like I drive an electric car. It charges while I sleep. It doesn't matter if an ICE can fuel in a few minutes.
"That's fair, you are absolutely right I should not have unscrewed the water hose and put it away before turning on the valve to water the plants, even though you specified that order, I should've used some common sense."
There is going to be so much pain from VLA malicious compliance. If the current gen of LLMs are anything to go by I can already see it being an hilariously massive problem. People are careless.
> it honestly doesn't matter if it takes a robot longer than a human to clean your house
it depends. in order for my slow ass Roomba to clean my floors, I have to move a bunch of things out of the way and not use the room.
on the flip side, i think this "the robot cannot automate the whole task" thing is reductive too. suffice it to say, EVERYTHING matters, there honestly nothing that "honestly doesn't matter."
The difference between gpt 2 and 3 was insane. 2 could generate limericks when it wasn't repeating a word 300x. 3 could actually do some things. By comparison gemini robotics has hardly changed at all.
I will also point out that slow, non-fluid robotics is on a totally different level of difficulty from fast fluid motion. Asimov could walk pretty smoothly, but it didn't fall over because it used a very careful sequence that was never unbalanced; you could pause at any point without falling over. Move faster, like boston dynamics, and you need to account for the change in balance from your arms swinging... or rather, you need to be able to account for the rotational inertia etc from moving multiple masses along complex paths with multiple points of articulation at hundreds or thousands of times per second.
An algorithm to fold tshirts 90% of the time is easy. The cloth hangs down by gravity and you can just look for right angles (corners), find their coordinates with binocular matching, and move them to meet each other. Getting 99%, or folding them quickly, so that the fabric is actually moving instead of just hanging still- incredibly, incredibly more complex.
> folding them quickly, so that the fabric is actually moving instead of just hanging still- incredibly, incredibly more complex.
TBF fabric is much more difficult to simulate than a bipedal body is.
As I recall openai had mujoco playing soccer nearly 10 years ago. Obviously real world bodies are much more difficult but I'd be curious to learn why that is.
Even being able to simply do the task isn't enough here. It also has to do the task while also never, ever, ever accidentally folding your toddler as though it were a towel. Having industrial-strength robots around inherently unpredictable humans in a fail-safe way is an extremely difficult problem that will not be addressed any time soon.
This is the first press videos I’ve seen of 1x speed that seems reasonable, imho. Though I see elsewhere they report only 50-75% success rate- not clear if that's first attempt, or total?
They're fluid, but they are slow, which is highly likely to be a safety thing, which will have the side effect of making many motions more difficult, not less.
LLMs aren't themselves hurting anyone, no matter how much certain people like to pretend otherwise, whereas an AI in a robot with significant motors in absolutely can and will. There are reasons industrial ones live in safety cages after all.
Expecting them to be like inverse kinematic driven digital dolls is wrong because the optimization won't be for matching that but something like "net reduce energy consumption" which for electric motors in multi joint arms will look a bit odd.
We don't get virtuous feedback loops in hardware though. Muscles are incredible engineering that took half a billion years. Token generation (speech) is low hundred thousands maybe. Unless of course improved AI navigates the space of ideas so well that it can give us alloys or synthetic flesh that does twitch as fast...
These robots are unusable in their current form. Slow and expensive, humans perform this work better and cheaper. Looks like every other robot before it, a bunch of servos and no brain.
Better to go slow and controlled. Giving physical action to a model is dangerous and must be carefully monitored. Slow is good. Oversight is important.
I've kind of given up on humanoid robotics, because of how bad the actuators are. There has been no innovation in robotic actuators since Honda's Asimo. There's just no way that someone wants a 80kg wobbling tin can in their home or workplace.
My bet is that the final robotic revolution will use genetically modified human/animal bodies with replaced brains. You'll have to stretch your ethics a bit, but if you grow a bear genetically modified in a way that it has no consciousness or thought, it'll make a much better construction worker than any humanoid robot. You'll just need to wire it up with neuralink and then control it via LLM. Fast animals can be used to deliver packages, and giraffes for warehouses.
I’m not even sure if you are joking there or haven’t thought your proposal through. Sure giraffes are tall, but they can barelly lift any weight. What use would a robotically controlled giraffe be in a warehouse?
> There has been no innovation in robotic actuators since Honda's Asimo.
I very much doubt this. If nothing else the MIT Cheetah’s actuators are a whole different ballgame compared to asimo’s actuators. (Backdriveability, variable stiffness) And then there is a lot of interesting work being done with combining elastic elements with the actuators.
Well, a giraffe could do jobs where you need to reach high, such as replacing a lightbulb. Sure, I'm half-joking.
Asimo used BLDC motors with strain-wave gearing, which is pretty much standard today on high-end humanoid robots. The only thing that has happened is that these are much cheaper today, and might be slightly more optimized.
this is a tangent but you should revisit your assumptions about giraffe anatomy. their necks actually weigh a tremendous amount and are kept upright by an interesting system whereby they actually stretch this giant tendon to bend down and the tendon's tension lifts it back up. super cool to learn about - they dissect one in a documentary series I can't think of just now.
Cortical Labs has been doing work in this area. They had a Doom demo a while ago.
I think it's a matter of tradeoffs. Perhaps you'd have biological version as a versatile all-around thing, but many tasks don't require that so likely the tin variants are more durable for many tasks. Also, in terms of reasoning, the brain bandwidth is very low and high latency so there are advantages to silicon chips there.
It's interesting to learn actuators are "behind". I kept seeing cool stuff in the 3D printing space and thought there's a lot of progress. I'd love to learn more.
That is a truly terrifying future to imagine. Fortunately I would hazard to guess the medical technology to accomplish something like this is very, very far away. Not only because of the raw technological challenge, but also the barriers to development that scientists, doctors, and engineers would face before even being allowed to conduct experiments.
I do agree though the humanoid form is a dead end for robots. Just build giant cubes that process inputs and give outputs, like a dishwasher. Why wash dishes with meat wand tentacles or try to recreate meat wand tentacles when you can accomplish the job in a wholly different way with far greater efficiency...?
Where's the clothes foldeing cube? Analogous to the clothes washer and clothes dryer.... the clothes folder...
Why stop at dishwashing...? Sell an entire integrated robotic kitchen.
I think the only real barriers are ethical. I'm pretty sure we'll figure out to do a completely synthetic biological robots without any kind of biological brain in the next 10-20 years, and they don't even have to resemble animals or humans. This should alleviate some of the ethical issues.
I could not disagree more with the claim about no improvement in robotic actuators.
The torque density and price of actuators has fallen dramatically since Ben Katz's MIT work on mini cheetah. The actuators on the Unitree G1 based on that work are powerful for their size and near quasi-direct-drive. The motors on the BD E-Atlas are completely passively cooled and appear to have really good torque density. Actuators have never been improving faster than they are now.
That seems like a problem orders of magnitude harder than making a humanoid robot. We haven't even figured out how to make hamburgers without cows at a marketable price yet.
> We haven't even figured out how to make hamburgers without cows
Isn't that OP's point? Engineering muscle and sinew is harder than coming up with the control software. The cheapest way to a robot thus emerges as just taking the natural stuff and adding an artificial brain to it versus trying to re-engineer the bones and muscles with metal and plastic.
The world can't support the number of bodies on it already, and you're proposing adding mindless meatbags that still need feeding and their waste managed? Are you going to accept an amazon package that's had giraffe poop sprayed on it while picking, or change your meat-robot's diaper? I didn't think so.
Can anyone that works on this technology provide an honest assessment of where this technology actually stands? How much instrumentation is actually required, what the interaction quality is, how much trouble do humanoids have with in the wild daily tasks like turning doorknobs, recovering from falls, avoiding knocking into things, etc.
I work in this field, and I wrote my bachelor's thesis here, not with humanoids but with VLAs (think chatgpt connected to a robot arm)
It's certainly not there yet for anything practical, there's also certain bits and structures that don't have accurate names during construction, and it is important to keep that in mind - so a robot is unlikely to understand what it means to say "put the left bit of this box onto this right bit" due to ambiguity, a human would understand that
Plus we have no good reliable accuracy testing data in most cases (most tests occur on a few demos, but that isn't a good representation of how must things work), popular benchmarks, such as libero have been saturated, and nearly everything gets 95% there, most companies and researchers have their own benchmarks here.
Plus companies lie alot, and do very dangerous things in thier videos, I.e. these robots should not be standing very close to humans, because of being dangerous.
There are also legitimate concerns of misuse of these robots that need to be accounted for, misuse does not have to be warfare, but can be as simple as confusing it while it is cutting tomatoes with a knife.
Turning doorknob is easy, and fail recovery is also being worked on, but we don't have reliable statistics anywhere on that. The hard part is on practical things, as in when placing bricks or attaching a part during manufacturing it needs to ensure that it is aligning everything correctly....and that's hard, while it is impressive, it is very irresponsible to keep humanoids at home (people are irresponsible when untrained), for example, lawnmowers injure about 6400 people a year...and that is not an everything machine.
Humanoids in general are...not appealing in specific, due to maintainable of joints, complexity, but robot arms in particular, expecially on wheels (check mobile aloha), are likely to be able to do tasks such as clean up in hotels, after a guest had left, or replace some cooks in restaurants (if their work is consistent)
IMHO the real test is if any robotic startup currently selling (or planning to sell) robots as a service for homes not just use it but gets returning users from it.
I did professionally few prototypes with robots and progress is real yet very far from what the average customer would find reliably useful in menial tasks.
FWIW I do think https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... remains relevant, namely dexterity is also a hardware problem, grippers aren't hands. They even clarify "multi-finger dexterous manipulation remains challenging." and those aren't even fingers with a lot of sensors.
There is a LOT of monkey business when it comes to robotics industry, a lot of demos you see are well crafted for that specific demo, a lot of times there’s even actual fraud “like claiming it’s an autonomous but it was actually teleoperated during the demo”, a lot robots that make most hypes are practically useless (dog robots or qugv for example) no matter how they try to create a usecase, when it comes to practicality, most fail. Robotics as an industry is niche (not to be confused with cobotics) the only exception is drones which is something I said in a workshop back in 2020 that drones are the only potential and will dominate if we solved the endurance issue, which is also why they immediately got weaponized because humans are usually evil and will utilize useful stuff in bad things.
Humanoids are far from being remotely useful in real life scenarios, yet.
Fun fact: most (if not all) qugv can’t go reverse on a stairway.
Today, humans convert their labor to capital. Capital holders need labor (humans) to acquire more capital. When the price of inference for these robots becomes less than the price of labor then capital holders don’t need labor.
Obviously, AI impacts non-manual labor too, but a significant portion of the world population does manual labor.
> Today, humans convert their labor to capital. Capital holders need labor (humans) to acquire more capital. When the price of inference for these robots becomes less than the price of labor then capital holders don’t need labor.
Ignoring quibbling about inference not being the only cost and other issues and just accepting the proposed end state: this is very good if capital is effectively democratized, and apocalyptically bad if it remains highly concentrated in a narrow class.
Not sure it is just about inference costs, errors in handling things and beings in the real world have a very different "surface" from immaterial applications - robots might need to have failsafes, independent limiters, etc. there.
Great point, inference is just one variable. It’d be really great to know how much those tasks costs in inference. I’m assuming they cost significantly higher than human labor. Therefore, the inference cost is so high that it makes the other variables seem irrelevant for the time being.
It's good that we'd be able to increase the amount of goods and services produced and reduce prices. For example, a lot of people would benefit from access to cheap, reliable heart surgery within a few days of learning they needed it. It would be good if the exponential growth that has elevated our wellbeing for the last millenia wasn't tethered to exponential population growth that would be disastrous if we even expected it to continue.
It would be bad to remove demand for humans from the economy, of course. Humans have inherent moral value, and so it's good that our current system gives them economic value as well. But there's more than one way to achieve that end, and the massive quantity on the good side of the scale suggests it may be worth investigating the others instead of opposing the advancement outright.
It's good for capital holders. It does imply a further shift in the distribution of wealth though with all the attendant implications thereof.
I question the premise that humanoid robots are "around the corner". I suspect this will turn out more like self-driving cars which are still a very slow burn.
A lot of these household tasks could be automated without the need for humanoid robots. Roombas can clean your floor, automated rubbish disposal can be integrated into houses and flats, grocery delivery and storage could be done via an automated intake, cataloguing, and storage system. Humanoids are just creepy and you can't really trust them.
Could you sleep easily knowing that you have one in your house?
What if you oppose the political views of its creators?
>automated rubbish disposal can be integrated into houses and flats, grocery delivery and storage could be done via an automated intake, cataloguing, and storage system
This is a complete non-starter for already-built apartment blocks, terraced homes, and even most semi-detached. It's an interesting but costly solution for new detached homes.
Humanoids are a useful form factor because the already-built human world is, definitionally, built for humanoids.
Doing what's ethically right rarely means taking the most convenient path. Fridges and pantries take a considerable amount of space in our homes. These could become part of an automated storage system.
we ve stopped building infrastructure but we have to do it. Rail, car , electricity, fiber infrastructure developed quickly, yet we refuse to even plan for robotic infrastructure. We can't be stuck forever in the 19th century
Except Roombas can only clean 80-90% of my floor, and not as well as a real vacuum. And we have a dog that leaves toys everywhere so probably it does more like 60% of the job when accounting for clearing its path, emptying it, etc… To get to 100% you basically need the whole human(oid).
> Humanoids are just creepy and you can't really trust them.
Ultimately I shouldn't have to trust my robots. If my roomba or my dishwasher go haywire they won't pinch my finger off. They physically can't listen to me or spy on me. These are good robots.
Why do we want to add more bodies on a finite plane of existence with an infinitely-expanding base (humans)?
We have a finite plane of existence (Earth), with an infinitely-expanding consumer base (humans), and we want to add robotic competition to that finite plane of existence? The data center AI is more appealing (I guess) since it doesn't compete literally shoulder-to-shoulder with me. Why do we want dexterous robots when we have humans?
We show many signs of stopping our infinite expansion.
As for why we want dexterous robots, the obvious answer is to do things that we don't want to do. Things that are back-breaking, disgusting, dangerous, or just plain boring.
Running a full LLM to actuate a robot is way too heavy, the minimum latency they can get down to is probably 1-2 seconds even with powerful GPUs, which is not very useful for practical robotics applications.
Machine vision should definitely be handled by ML, but motion actuation should be relegated in the realm of traditional PID style linear/nonlinear control. Again, the tech is cool, but the practical usefulness of using a full LLM as a controller will probably run into hardware limitations.
On latency: that is true now but likely false in the near future. There is so much potential left over for optimizing inference further.
On PID: the field has been stuck trying to do analytical/optimization-based control for decades, and end-to-end control has shown incredible performance (e.g., SoTA cost of transport in legged locomotion) and robustness (e.g., not falling over when stepping on a pile of leaves) - while being far scalable (in terms of how fast it is to get a new robot up and running). Which is not to say it's perfect but it seems like it's a step in the right direction.
Better yet, companies like Physical Intelligence are doing good with hierarchical ("fast-slow") architecture to address both the intelligence and latency fronts.
10Hz is still a 0.1s of latency, which doesn't seem like a lot until you realize that an Arduino typically runs at 16 Mhz, which is literally a million times faster than running a full LLM.
So, yeah, for complex reasoning and sensory processing, LLMs are the correct choice, and Gemini is especially strong at spatial reasoning over ChatGPT/Claude. But for actual motion/actuation? LLMs are the wrong tool for the job, probably easier to have LLMs program a reusable workflow in a script for repeated tasks instead of invoking LLMs after the first time.
I don’t think this is where the robotics revolution is going to happen. The big one will be using AI to design and build highly specialized robots to automate mining and manufacturing.
Big humanoid robots are expensive, the actuators still suck, and the control problems are hard. Their big advantage is that they can occupy human-shaped spaces, but struggle to do anything useful.
Not to say we won't get humanoid robots eventually, but I think there's probably some low hanging fruit for people to make some other kinds of solutions. Specialized robots for industrial environment, well-thought out appliances for the home.
It would be a bit surprising if the progression was Roomba -> humanoid robot.
I hope humanoids will be ever present, it’s just that the scale of impact from specialized robots will be far greater. Humanoids will be our butlers, but the other robots will be creating a world of unimaginable abundance.
No mention of whether the robot's models run locally or in the cloud. Ie. Does it require an internet connection?
For a robot to be in my house it would have to run locally, there's no way I'm allowing one that runs in the cloud to operate my washing machine for example.
> Many robotic applications need to operate without network latency or internet connectivity. Gemini Robotics On-Device 2 is built specifically to handle these constraints — it is our most-efficient vision-language-action model (VLA) optimized to run locally on robotic devices.
The only thing that bothers me is what happens when robots finally automated "all the mundane day-to-day tasks" including our jobs, what is there left to do for common folks who are not geniuses working at Google/Anthropic/ChatGPT or occupy the C-Suite of these companies?
The question of agency and the question of income are different. The former - we do things manually that can be automated for fun all the time and make entirely arbitrary weird things to do for no value. However some find it hard to move from work to leisure.
Relevant: A San Francisco company is advertising humanoid robot housecleaning services[1]. However those 'bots require local supervision and a human controller watching through its video feed.
Presumably that company could use the Gemini Robotics product to eliminate the remote controller.
I actually like Tau's videos a lot better than Google's. An agile robot getting out of a car carrying a tote is actually novel and difficult and useful. In contrast, what Google is showing here is mostly glorified pick and place with low success rates on a needlessly complex robot and a voice LLM slapped on top to pre-announce its moves like an anime character, with fancy video editing and bubbly music to distract you from how slow it is.
Sure Tau is teleoperated, but teleoperating an agile movement like that is actually really hard to get right and still involves AI to keep the robot balanced. Tau is a lot closer to real deployment than Google, and when it performs useful tasks it is simultaneously collecting the data to eventually automate those tasks.
A 36% success rate on screwing in a light bulb means there's probably something to LeCunn's take that VLM/VLA models aren't going to be the thing powering tomorrow's robots, but only time will tell.
I think that people will argue around that. Self-driving cars are already strong enough to kill people and even if you only made the robot strong enough to lift 15 pounds of weight, that's probably strong enough to slit someone's throat with a kitchen knife while they sleep... There probably isn't a level of strength that's weak enough to be completely safe but strong enough to be useful.
So at some point you have to trust that the tech is safe. Both in terms of "robot won't go off the rails", but also in terms of hostile actors can't remotely take over your robot while you sleep.
In terms of sleeping, personally, I would like for the law to mandate that robots must have a physical off switch, in a very visible location, that physically disconnects power. The switch should be illuminated while in the ON position. What makes me a bit pessimistic there is that we don't even have laws to mandate webcam indicator lights (e.g. a very tiny red LED) must be ON in hardware.
I loved the video. Also, I had a lot of fun watching it while imagining I was living in 2051 and this whole progression had ran its course. It's impossible to predict the future, but certain events can negate whole branches of possibility. This one video in particular prunes the "we never figured out robotic motion and thus all the science fiction scenarios built on that premise never came to happen" subtree.
I believe that LLMs represent the future of robotics in this field. Of course, I don’t want a robot uprising, but if our goal is simply to increase the utility of robots in manufacturing (within the necessary constraints), then LLMs are the only powerful path forward.
Humanoid robots can be dropped in wherever human labor is currently used.
That makes them extremely easy to sell to industry. This unlocks, in principle, a large chunk of the potential untapped industrial automation market that still relies on human labor because the ROI of redesigning production lines didn't make sense.
These humanoid robots would cost $10-100k to make and would require costly maintenance for them to do the job people in poor countries do for a dollar. I don't say we won't come to it eventually, but not in near future
More realistically it seems that llms in several years could help dramatically decrease costs of automation and make it available for more industries
ABB, EPSON, FANUC, Kawasaki... Virtually every cars in past 50-60 years have had robots and some form of AI involved in assembly. Some of them already support standard humanoid/dogbot tools in their products.
Humanoid startup execs just don't talk about that.
And absolutely nothing in your company charter says the military won't be allowed to buy it. Y'all think we've never seen a Terminator movie before, huh.
I do think robotics would come up with more safety mechanisms (provably safe motion planning etc) just because the risk is a lot more serious than LLMs spitting half-truths
> It can better detect when humans are nearby, trigger safety tool calls and bring the robot to a safe stop if someone approaches too closely.
If the robots stop when humans are too close, wouldn't that mean that robots for close interaction or handling of humans need a whole other level of control?
Is the fundamental challenge for AI directed movement in 3D space the continuity of the environment? Vs LLM's which train on discrete and limited token sets.
Can someone that uses these models provide some tips or docs on how best to get to use these at home to test around? Is the best thing a virtual environment?
Given the rapid advancements in AI. It is not what we thought it will, what it is, what we didn't wanted it to be, but it is great for being what it is.
What's the point of "releasing it"? It only makes sense in the labs, and what would calling your APIs give "me" as a researcher, other than a baseline to beat? mah
the model is real, but it feels 100% internal
Gemini is the only LLM I've ever experienced that decided on its own to tell me that something was a bad idea. Which, ok, in itself it's not a bad thing, we actually need more of that instead of the pathological sycophancy that all current models suffer from.
But in a physical robot? Yeah, that thing is going to punch me in the face, eventually. Or worse.
I use Sonnet and Opus everyday. Believe me, it's very common they tell me that something is bad idea, write a rationale and suggest better solutions.
Gemini doesn't remember almost anything after 2-3 follow ups. I have to paste the same "system prompt" at the top of each message and it still doesn't understand it.
It's funny how Google is essentially trying to compete with the software side of Tesla only - Waymos (which AFAIK they plan to partner with major automakers), now this, etc.
Another way to look at it is that they're avoiding some of the expensive parts of making products that use AI, while leaning into their strengths. No Tesla required within the explanation.
Seems smart. There are dozens of companies that make low-margin cars. There's only one company that has managed to make a working autonomous software driver.
For both manipulation and autonomous driving, google has invested in approaches with custom hardware, and off-the-shelf hardware, and a blend (which is what waymo is).
I'm a researcher at Deepmind that contributed to these models. (And the opinions here are my own)
Just want to say, Deepmind is a great place to work and the only (Edit: one the few unique labs!) lab where you can move from large frontier models (Gemini), frontier open models (Gemma), robotics (what you see here), science (weather, biology, more) and basically any other topic related to intelligence. It's really an incredible place to be, with incredible people. Consider joining! And thank you for the enthusiasm here.
I’m a mechanical design engineer (10 years) turned software engineer (7 years in autonomous vehicles). Is there a particular role you would recommend for someone with my experience?
How are you engaging with the fundamental problems explained in this blog post https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex...
- the relatively crude tactile and proprioceptive sensing apparatuses of robots when compared to humans
- the limited availability of multisensory, perception-action coupled training data
Genuinely curious!
Is DeepMind basically Google innovation hub ???
What happens when those ideas need to be scaled into real products, though? For instance, I can't really imagine Google being fully committed to manufacturing and selling robotic arms at scale.
So glad to see that you are still involved and have your inspiring enthusiasm! Meeting you was a huge inflection point in my journey with ML/AI.
Would you mind expanding on the difference between frontier model development and open model development? I was under the impression that they were nearly the same thing and would thus be handled by the same team. My impression is that open models tend to go through more safety testing and are generally smaller, but are otherwise the same thing. Shouldn't most of what is commonly referred to as 'AI research' be applicable to both?
Congrats! Awesome release! (can i have access)
Join the waitlist! It's in the blogpost
Hey Ravin, I’m executive director for AGI Society which has run the AGI conference (https://agi-conference.org/) since 2007
We’d love to invite you to speak about your work next year at the AGI Conference if you’re open to it.
Alexander Lerchner spoke this year and was great.
I’ll reach out on email if you have a preferred one, it was unclear on your site which email you prefer.
Good stuff, yay, good.
I do want to pick a nit in this one,
> and the only lab where
I do think Allen Institution for AI (AI2) is has coverage across most of these domains ( albeit their frontier isn't nearly so far out, hopefully the $152m NSF awarded them + Nvidia is a fruitful partnership there).
For example, robotics: MolmoBot, MolmoSpaces, MolmoAct, https://allenai.org/embodied-ai
Totally fair and thank you for nit picking! I edited my comment. AI2 is doing amazing stuff, I read their papers and code all the time and appreciate how open they are.
What are you doing to prevent abuses by governments, corporations and militaries? (non snarky question)
How much of this technology is going to benefit and uplift the Common Peasant and how is going to be used for increasing surveillance and control?
While Anthropic and Open AI get 80% of the attention here, it's impressive to see how much Google is doing: near frontier model, fast models, open weight models, image generation, video generation, music generation, robotics, etc.
It's certainly an odd situation where Google is on the forefront of all these emerging technologies while the latest chatbots from OpenAI and Anthropic are getting breathless press releases and wildly hyperbolic coverage ("rogue AIs" "too dangerous, must be banned" etc...)
Google acting like a normal but competent company that's just chugging away, meanwhile the hype cycle is propping up its upstart competitors to truly ludicrous valuations.
> Google acting like a normal but competent company that's just chugging away
Competent is a stretch. Google's AI offering seems to be, once again, PM led–lots of constantly-changing brands being merged and deprecated with zero customer support or service.
They'll almost certainly be one of the survivors. Their technical competece is unmatched. But that doesn't mean they have a great product in the way both OpenAI and Anthropic do. (Outside their datacentres, which are a legitimate feat.)
> Google acting like a normal but competent company that's just chugging away,
Chugging away not releasing products or releasing ones worse than their competition typically. Not sure that’s what a normal competent company should be doing.
Google wins on infrastructure and creativity (NotebookLM, etc) but entirely lacks product coherence. The ecosystem is a mess, just like Microsoft's offering, and I don't think this is a coincidence but rather a symptom of the type of company in which these two companies have evolved.
Furthermore, Gemini is just all around a less trustworthy and mature model, for many reasons. Very smart but lacking the precision and holistically exhibited in the more recent models from OpenAI and Anthropic. On the flip side, Google's work on Gemma is unmatched.
Google DeepMind was at the forefront of AI, and they are currently developing new concepts while others chase benchmark percentages.
And protein folding + Nobel Prize
Google DeepMind disbands its Nobel-prize winning AlphaFold team
https://www.engadget.com/2225849/google-shuts-down-alphafold...
https://news.ycombinator.com/item?id=49098375
Baffles me how this is not that well known
And AlphaGo Zero .. probably the first real-world deployment of a self-improving system that beat humans.
These robots look slow and not very fluid in their motions, but LLMs like ChatGPT also looked very dumb initially. If progress is as fast as LLMs , this could have massive applications in a few years.
For house keeping tasks the robotics companies love showing off... it honestly doesn't matter if it takes a robot longer than a human to clean your house. As long as it gets done before you get home from work, no big deal.
I'm bearish that they'll be economically viable in the house for a very long time.
For businesses the bar for adoption is very low: If the thing can work repetitive jobs for 24 hours a day and replace 3 shifts, the purchase bar is nominally anything less than 3 x human salary if your budgeting horizon is 1 year. That's a high number, and probably fairly easy to achieve.
For homes, it's a very different bar. You'd have a hard time convincing most American families to purchase anything with a >$1000 price tag. Currently that's pretty much impossible for a humanoid.
Yeah, the dishwasher takes much longer to do the dishes than I do by hand, but that's cool. That said, the dishwasher is out of the way.
I don't think we'll get household robots anytime soon. Hell, the only ones that can afford them will be the same that would hire human household help.
Exactly. People buy those automated circular floor vacuums. Anyone can get it done with a regular broom/vacuum in a small fraction of the time it takes for those things. Just like I drive an electric car. It charges while I sleep. It doesn't matter if an ICE can fuel in a few minutes.
We all know sexbots will be the real killer app
"That's fair, you are absolutely right I should not have unscrewed the water hose and put it away before turning on the valve to water the plants, even though you specified that order, I should've used some common sense."
There is going to be so much pain from VLA malicious compliance. If the current gen of LLMs are anything to go by I can already see it being an hilariously massive problem. People are careless.
> it honestly doesn't matter if it takes a robot longer than a human to clean your house
it depends. in order for my slow ass Roomba to clean my floors, I have to move a bunch of things out of the way and not use the room.
on the flip side, i think this "the robot cannot automate the whole task" thing is reductive too. suffice it to say, EVERYTHING matters, there honestly nothing that "honestly doesn't matter."
The time between gpt 2 and 3 was 15 months. The time between gemini robotics 1 and 2 was also 15 months: https://blog.google/products-and-platforms/products/gemini/h...
The difference between gpt 2 and 3 was insane. 2 could generate limericks when it wasn't repeating a word 300x. 3 could actually do some things. By comparison gemini robotics has hardly changed at all.
I will also point out that slow, non-fluid robotics is on a totally different level of difficulty from fast fluid motion. Asimov could walk pretty smoothly, but it didn't fall over because it used a very careful sequence that was never unbalanced; you could pause at any point without falling over. Move faster, like boston dynamics, and you need to account for the change in balance from your arms swinging... or rather, you need to be able to account for the rotational inertia etc from moving multiple masses along complex paths with multiple points of articulation at hundreds or thousands of times per second.
An algorithm to fold tshirts 90% of the time is easy. The cloth hangs down by gravity and you can just look for right angles (corners), find their coordinates with binocular matching, and move them to meet each other. Getting 99%, or folding them quickly, so that the fabric is actually moving instead of just hanging still- incredibly, incredibly more complex.
> folding them quickly, so that the fabric is actually moving instead of just hanging still- incredibly, incredibly more complex.
TBF fabric is much more difficult to simulate than a bipedal body is.
As I recall openai had mujoco playing soccer nearly 10 years ago. Obviously real world bodies are much more difficult but I'd be curious to learn why that is.
Even being able to simply do the task isn't enough here. It also has to do the task while also never, ever, ever accidentally folding your toddler as though it were a towel. Having industrial-strength robots around inherently unpredictable humans in a fail-safe way is an extremely difficult problem that will not be addressed any time soon.
This is the first press videos I’ve seen of 1x speed that seems reasonable, imho. Though I see elsewhere they report only 50-75% success rate- not clear if that's first attempt, or total?
Gotta admit google is pretty ballsy for directly attacking Moravec's Paradox: https://en.wikipedia.org/wiki/Moravec%27s_paradox
They're fluid, but they are slow, which is highly likely to be a safety thing, which will have the side effect of making many motions more difficult, not less.
LLMs aren't themselves hurting anyone, no matter how much certain people like to pretend otherwise, whereas an AI in a robot with significant motors in absolutely can and will. There are reasons industrial ones live in safety cages after all.
Expecting them to be like inverse kinematic driven digital dolls is wrong because the optimization won't be for matching that but something like "net reduce energy consumption" which for electric motors in multi joint arms will look a bit odd.
We don't get virtuous feedback loops in hardware though. Muscles are incredible engineering that took half a billion years. Token generation (speech) is low hundred thousands maybe. Unless of course improved AI navigates the space of ideas so well that it can give us alloys or synthetic flesh that does twitch as fast...
These robots are unusable in their current form. Slow and expensive, humans perform this work better and cheaper. Looks like every other robot before it, a bunch of servos and no brain.
> If progress is as fast as LLMs
Is this the new "x technology is a year away" ?
I believe that it's also for safety, these robots would easily destroy themselves if not restrained (and cause damage).
Better to go slow and controlled. Giving physical action to a model is dangerous and must be carefully monitored. Slow is good. Oversight is important.
yes at 1k tok/sec i think these robots will be fast and very fluid.
I've kind of given up on humanoid robotics, because of how bad the actuators are. There has been no innovation in robotic actuators since Honda's Asimo. There's just no way that someone wants a 80kg wobbling tin can in their home or workplace.
My bet is that the final robotic revolution will use genetically modified human/animal bodies with replaced brains. You'll have to stretch your ethics a bit, but if you grow a bear genetically modified in a way that it has no consciousness or thought, it'll make a much better construction worker than any humanoid robot. You'll just need to wire it up with neuralink and then control it via LLM. Fast animals can be used to deliver packages, and giraffes for warehouses.
> giraffes for warehouses.
I’m not even sure if you are joking there or haven’t thought your proposal through. Sure giraffes are tall, but they can barelly lift any weight. What use would a robotically controlled giraffe be in a warehouse?
> There has been no innovation in robotic actuators since Honda's Asimo.
I very much doubt this. If nothing else the MIT Cheetah’s actuators are a whole different ballgame compared to asimo’s actuators. (Backdriveability, variable stiffness) And then there is a lot of interesting work being done with combining elastic elements with the actuators.
Well, a giraffe could do jobs where you need to reach high, such as replacing a lightbulb. Sure, I'm half-joking.
Asimo used BLDC motors with strain-wave gearing, which is pretty much standard today on high-end humanoid robots. The only thing that has happened is that these are much cheaper today, and might be slightly more optimized.
this is a tangent but you should revisit your assumptions about giraffe anatomy. their necks actually weigh a tremendous amount and are kept upright by an interesting system whereby they actually stretch this giant tendon to bend down and the tendon's tension lifts it back up. super cool to learn about - they dissect one in a documentary series I can't think of just now.
Cortical Labs has been doing work in this area. They had a Doom demo a while ago. I think it's a matter of tradeoffs. Perhaps you'd have biological version as a versatile all-around thing, but many tasks don't require that so likely the tin variants are more durable for many tasks. Also, in terms of reasoning, the brain bandwidth is very low and high latency so there are advantages to silicon chips there.
It's interesting to learn actuators are "behind". I kept seeing cool stuff in the 3D printing space and thought there's a lot of progress. I'd love to learn more.
I think they're doing it the other way around, i.e. growing a biological brain for the AI, which would then be used in a robot.
That is a truly terrifying future to imagine. Fortunately I would hazard to guess the medical technology to accomplish something like this is very, very far away. Not only because of the raw technological challenge, but also the barriers to development that scientists, doctors, and engineers would face before even being allowed to conduct experiments.
I do agree though the humanoid form is a dead end for robots. Just build giant cubes that process inputs and give outputs, like a dishwasher. Why wash dishes with meat wand tentacles or try to recreate meat wand tentacles when you can accomplish the job in a wholly different way with far greater efficiency...?
Where's the clothes foldeing cube? Analogous to the clothes washer and clothes dryer.... the clothes folder...
Why stop at dishwashing...? Sell an entire integrated robotic kitchen.
I think the only real barriers are ethical. I'm pretty sure we'll figure out to do a completely synthetic biological robots without any kind of biological brain in the next 10-20 years, and they don't even have to resemble animals or humans. This should alleviate some of the ethical issues.
I could not disagree more with the claim about no improvement in robotic actuators.
The torque density and price of actuators has fallen dramatically since Ben Katz's MIT work on mini cheetah. The actuators on the Unitree G1 based on that work are powerful for their size and near quasi-direct-drive. The motors on the BD E-Atlas are completely passively cooled and appear to have really good torque density. Actuators have never been improving faster than they are now.
That seems like a problem orders of magnitude harder than making a humanoid robot. We haven't even figured out how to make hamburgers without cows at a marketable price yet.
> We haven't even figured out how to make hamburgers without cows
Isn't that OP's point? Engineering muscle and sinew is harder than coming up with the control software. The cheapest way to a robot thus emerges as just taking the natural stuff and adding an artificial brain to it versus trying to re-engineer the bones and muscles with metal and plastic.
Did you look at 1X's artificial tendon driven hands?
The world can't support the number of bodies on it already, and you're proposing adding mindless meatbags that still need feeding and their waste managed? Are you going to accept an amazon package that's had giraffe poop sprayed on it while picking, or change your meat-robot's diaper? I didn't think so.
Can anyone that works on this technology provide an honest assessment of where this technology actually stands? How much instrumentation is actually required, what the interaction quality is, how much trouble do humanoids have with in the wild daily tasks like turning doorknobs, recovering from falls, avoiding knocking into things, etc.
I work in this field, and I wrote my bachelor's thesis here, not with humanoids but with VLAs (think chatgpt connected to a robot arm)
It's certainly not there yet for anything practical, there's also certain bits and structures that don't have accurate names during construction, and it is important to keep that in mind - so a robot is unlikely to understand what it means to say "put the left bit of this box onto this right bit" due to ambiguity, a human would understand that
Plus we have no good reliable accuracy testing data in most cases (most tests occur on a few demos, but that isn't a good representation of how must things work), popular benchmarks, such as libero have been saturated, and nearly everything gets 95% there, most companies and researchers have their own benchmarks here.
Plus companies lie alot, and do very dangerous things in thier videos, I.e. these robots should not be standing very close to humans, because of being dangerous.
There are also legitimate concerns of misuse of these robots that need to be accounted for, misuse does not have to be warfare, but can be as simple as confusing it while it is cutting tomatoes with a knife.
Turning doorknob is easy, and fail recovery is also being worked on, but we don't have reliable statistics anywhere on that. The hard part is on practical things, as in when placing bricks or attaching a part during manufacturing it needs to ensure that it is aligning everything correctly....and that's hard, while it is impressive, it is very irresponsible to keep humanoids at home (people are irresponsible when untrained), for example, lawnmowers injure about 6400 people a year...and that is not an everything machine.
Humanoids in general are...not appealing in specific, due to maintainable of joints, complexity, but robot arms in particular, expecially on wheels (check mobile aloha), are likely to be able to do tasks such as clean up in hotels, after a guest had left, or replace some cooks in restaurants (if their work is consistent)
It’s still at GPT-1 level, but GPT-2 moment feels imminent.
What signs point to an inevitable GPT-2 moment in this field?
IMHO the real test is if any robotic startup currently selling (or planning to sell) robots as a service for homes not just use it but gets returning users from it.
I did professionally few prototypes with robots and progress is real yet very far from what the average customer would find reliably useful in menial tasks.
FWIW I do think https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... remains relevant, namely dexterity is also a hardware problem, grippers aren't hands. They even clarify "multi-finger dexterous manipulation remains challenging." and those aren't even fingers with a lot of sensors.
There is a LOT of monkey business when it comes to robotics industry, a lot of demos you see are well crafted for that specific demo, a lot of times there’s even actual fraud “like claiming it’s an autonomous but it was actually teleoperated during the demo”, a lot robots that make most hypes are practically useless (dog robots or qugv for example) no matter how they try to create a usecase, when it comes to practicality, most fail. Robotics as an industry is niche (not to be confused with cobotics) the only exception is drones which is something I said in a workshop back in 2020 that drones are the only potential and will dominate if we solved the endurance issue, which is also why they immediately got weaponized because humans are usually evil and will utilize useful stuff in bad things.
Humanoids are far from being remotely useful in real life scenarios, yet.
Fun fact: most (if not all) qugv can’t go reverse on a stairway.
Robotics with AI is the real revolution.
Today, humans convert their labor to capital. Capital holders need labor (humans) to acquire more capital. When the price of inference for these robots becomes less than the price of labor then capital holders don’t need labor.
Obviously, AI impacts non-manual labor too, but a significant portion of the world population does manual labor.
> Today, humans convert their labor to capital. Capital holders need labor (humans) to acquire more capital. When the price of inference for these robots becomes less than the price of labor then capital holders don’t need labor.
Ignoring quibbling about inference not being the only cost and other issues and just accepting the proposed end state: this is very good if capital is effectively democratized, and apocalyptically bad if it remains highly concentrated in a narrow class.
> this is very good if capital is effectively democratized
Are there any indications to think it's possible in our world?
Not sure it is just about inference costs, errors in handling things and beings in the real world have a very different "surface" from immaterial applications - robots might need to have failsafes, independent limiters, etc. there.
Great point, inference is just one variable. It’d be really great to know how much those tasks costs in inference. I’m assuming they cost significantly higher than human labor. Therefore, the inference cost is so high that it makes the other variables seem irrelevant for the time being.
and this is good?
It's good that we'd be able to increase the amount of goods and services produced and reduce prices. For example, a lot of people would benefit from access to cheap, reliable heart surgery within a few days of learning they needed it. It would be good if the exponential growth that has elevated our wellbeing for the last millenia wasn't tethered to exponential population growth that would be disastrous if we even expected it to continue.
It would be bad to remove demand for humans from the economy, of course. Humans have inherent moral value, and so it's good that our current system gives them economic value as well. But there's more than one way to achieve that end, and the massive quantity on the good side of the scale suggests it may be worth investigating the others instead of opposing the advancement outright.
If all goes well, imagine a world in which you work because you want to not because you have to.
It's good for capital holders. It does imply a further shift in the distribution of wealth though with all the attendant implications thereof.
I question the premise that humanoid robots are "around the corner". I suspect this will turn out more like self-driving cars which are still a very slow burn.
As a 47-year-old man with no kids, I have to admit I'm looking forward to the possibility of having a robot help take care of me in my old age.
The fact that Star Wars like droids seem to be likely within our lifetime is WILD. It's crazy how normal everyone is acting about this.
As a person in my 50's with a parent in her 80's, this would be incredibly useful right now. And I've got many, many friends in similar situations.
Fucking hell, hiring a robot to care after an elderly parent. Now that's some disturbing shit.
A lot of these household tasks could be automated without the need for humanoid robots. Roombas can clean your floor, automated rubbish disposal can be integrated into houses and flats, grocery delivery and storage could be done via an automated intake, cataloguing, and storage system. Humanoids are just creepy and you can't really trust them.
Could you sleep easily knowing that you have one in your house? What if you oppose the political views of its creators?
>automated rubbish disposal can be integrated into houses and flats, grocery delivery and storage could be done via an automated intake, cataloguing, and storage system
This is a complete non-starter for already-built apartment blocks, terraced homes, and even most semi-detached. It's an interesting but costly solution for new detached homes.
Humanoids are a useful form factor because the already-built human world is, definitionally, built for humanoids.
Doing what's ethically right rarely means taking the most convenient path. Fridges and pantries take a considerable amount of space in our homes. These could become part of an automated storage system.
we ve stopped building infrastructure but we have to do it. Rail, car , electricity, fiber infrastructure developed quickly, yet we refuse to even plan for robotic infrastructure. We can't be stuck forever in the 19th century
Except Roombas can only clean 80-90% of my floor, and not as well as a real vacuum. And we have a dog that leaves toys everywhere so probably it does more like 60% of the job when accounting for clearing its path, emptying it, etc… To get to 100% you basically need the whole human(oid).
> Humanoids are just creepy and you can't really trust them.
Ultimately I shouldn't have to trust my robots. If my roomba or my dishwasher go haywire they won't pinch my finger off. They physically can't listen to me or spy on me. These are good robots.
https://www.technologyreview.com/2022/12/19/1065306/roomba-i...
Why do we want to add more bodies on a finite plane of existence with an infinitely-expanding base (humans)?
We have a finite plane of existence (Earth), with an infinitely-expanding consumer base (humans), and we want to add robotic competition to that finite plane of existence? The data center AI is more appealing (I guess) since it doesn't compete literally shoulder-to-shoulder with me. Why do we want dexterous robots when we have humans?
Is this a joke? To do work that we don't want or have time to do, obviously.
Human time is the most scarce resource on earth, and we will never invent a way to have more than 24 hours in a day.
We show many signs of stopping our infinite expansion.
As for why we want dexterous robots, the obvious answer is to do things that we don't want to do. Things that are back-breaking, disgusting, dangerous, or just plain boring.
Running a full LLM to actuate a robot is way too heavy, the minimum latency they can get down to is probably 1-2 seconds even with powerful GPUs, which is not very useful for practical robotics applications.
Machine vision should definitely be handled by ML, but motion actuation should be relegated in the realm of traditional PID style linear/nonlinear control. Again, the tech is cool, but the practical usefulness of using a full LLM as a controller will probably run into hardware limitations.
On latency: that is true now but likely false in the near future. There is so much potential left over for optimizing inference further.
On PID: the field has been stuck trying to do analytical/optimization-based control for decades, and end-to-end control has shown incredible performance (e.g., SoTA cost of transport in legged locomotion) and robustness (e.g., not falling over when stepping on a pile of leaves) - while being far scalable (in terms of how fast it is to get a new robot up and running). Which is not to say it's perfect but it seems like it's a step in the right direction.
Better yet, companies like Physical Intelligence are doing good with hierarchical ("fast-slow") architecture to address both the intelligence and latency fronts.
AFAIK NVidia has a VLA running on their chips with 2B params at 10Hz. They’ve openly published a .5B model running at 10Hz
10Hz is still a 0.1s of latency, which doesn't seem like a lot until you realize that an Arduino typically runs at 16 Mhz, which is literally a million times faster than running a full LLM.
So, yeah, for complex reasoning and sensory processing, LLMs are the correct choice, and Gemini is especially strong at spatial reasoning over ChatGPT/Claude. But for actual motion/actuation? LLMs are the wrong tool for the job, probably easier to have LLMs program a reusable workflow in a script for repeated tasks instead of invoking LLMs after the first time.
I don’t think this is where the robotics revolution is going to happen. The big one will be using AI to design and build highly specialized robots to automate mining and manufacturing.
The AI will need generalized robots to build the highly specialized ones.
Why would building specialized robots require generalized robots?
Seems to me you'd do better if you built the robots with robots that were specialized in building robots.
Why not both?
Big humanoid robots are expensive, the actuators still suck, and the control problems are hard. Their big advantage is that they can occupy human-shaped spaces, but struggle to do anything useful.
Not to say we won't get humanoid robots eventually, but I think there's probably some low hanging fruit for people to make some other kinds of solutions. Specialized robots for industrial environment, well-thought out appliances for the home.
It would be a bit surprising if the progression was Roomba -> humanoid robot.
I hope humanoids will be ever present, it’s just that the scale of impact from specialized robots will be far greater. Humanoids will be our butlers, but the other robots will be creating a world of unimaginable abundance.
agriculture first (mining and manufacturing are already robotizing).
No mention of whether the robot's models run locally or in the cloud. Ie. Does it require an internet connection?
For a robot to be in my house it would have to run locally, there's no way I'm allowing one that runs in the cloud to operate my washing machine for example.
It does mention it though?
> Many robotic applications need to operate without network latency or internet connectivity. Gemini Robotics On-Device 2 is built specifically to handle these constraints — it is our most-efficient vision-language-action model (VLA) optimized to run locally on robotic devices.
This is incredible. Seems like a big achievement.
The only thing that bothers me is what happens when robots finally automated "all the mundane day-to-day tasks" including our jobs, what is there left to do for common folks who are not geniuses working at Google/Anthropic/ChatGPT or occupy the C-Suite of these companies?
The question of agency and the question of income are different. The former - we do things manually that can be automated for fun all the time and make entirely arbitrary weird things to do for no value. However some find it hard to move from work to leisure.
For income - who knows?
Relevant: A San Francisco company is advertising humanoid robot housecleaning services[1]. However those 'bots require local supervision and a human controller watching through its video feed.
Presumably that company could use the Gemini Robotics product to eliminate the remote controller.
[1] https://www.tau-robotics.com/
I actually like Tau's videos a lot better than Google's. An agile robot getting out of a car carrying a tote is actually novel and difficult and useful. In contrast, what Google is showing here is mostly glorified pick and place with low success rates on a needlessly complex robot and a voice LLM slapped on top to pre-announce its moves like an anime character, with fancy video editing and bubbly music to distract you from how slow it is.
Sure Tau is teleoperated, but teleoperating an agile movement like that is actually really hard to get right and still involves AI to keep the robot balanced. Tau is a lot closer to real deployment than Google, and when it performs useful tasks it is simultaneously collecting the data to eventually automate those tasks.
All VLM, VLA models. I wonder if these architectures will reach prod before Yann LeCunn's world model JEPA startup even gets off the ground in Europe.
Welsh Labs has a great video on JEPA and robotics. https://www.youtube.com/watch?v=kYkIdXwW2AE
A 36% success rate on screwing in a light bulb means there's probably something to LeCunn's take that VLM/VLA models aren't going to be the thing powering tomorrow's robots, but only time will tell.
It's still very early days. There are many benchmarks that LLMs scored 36% on just 18 months ago that they're now at 100% on.
Where did you get this perf metric? I couldn't find it in the release
I'm asked to prove I'm not a bot if I want to watch the top video with sound.
it s a trick question, they are actually looking for robots
Paint it yellow and ask why it is hiding a nuclear battery
I think the biggest snag with home robots will be that if it is strong enough to install your air conditioners, it's strong enough to break your ribs.
So the robots will need to be weak, so even weak people can overpower it. But then it loses a ton of it's most promising abilities.
I don't mean this like a "rogue robot" situation. I mean it like the robot gets confused, or someone sees a walking $10M lawsuit in their home.
I think that people will argue around that. Self-driving cars are already strong enough to kill people and even if you only made the robot strong enough to lift 15 pounds of weight, that's probably strong enough to slit someone's throat with a kitchen knife while they sleep... There probably isn't a level of strength that's weak enough to be completely safe but strong enough to be useful.
So at some point you have to trust that the tech is safe. Both in terms of "robot won't go off the rails", but also in terms of hostile actors can't remotely take over your robot while you sleep.
In terms of sleeping, personally, I would like for the law to mandate that robots must have a physical off switch, in a very visible location, that physically disconnects power. The switch should be illuminated while in the ON position. What makes me a bit pessimistic there is that we don't even have laws to mandate webcam indicator lights (e.g. a very tiny red LED) must be ON in hardware.
If they can't even get LLM/agentic coding right, I have little hope for a literal intelligent robot coming out of Gemini... but I wish them luck.
I loved the video. Also, I had a lot of fun watching it while imagining I was living in 2051 and this whole progression had ran its course. It's impossible to predict the future, but certain events can negate whole branches of possibility. This one video in particular prunes the "we never figured out robotic motion and thus all the science fiction scenarios built on that premise never came to happen" subtree.
Fingers crossed this inspires a Chinese lab to release some weights so people can benefit.
They already do: https://qwen.ai/blog?id=qwen-robotsuite
Nvidia also releases their Cosmos series models.
Did qwen actually release those robotsuite weights? I can't seem to find them anywhere.
I believe that LLMs represent the future of robotics in this field. Of course, I don’t want a robot uprising, but if our goal is simply to increase the utility of robots in manufacturing (within the necessary constraints), then LLMs are the only powerful path forward.
Is anyone building non-humanoid robots? Feels like humanoid is limiting the creativity of building a truly useful robot in today's age
Humanoid robots can be dropped in wherever human labor is currently used.
That makes them extremely easy to sell to industry. This unlocks, in principle, a large chunk of the potential untapped industrial automation market that still relies on human labor because the ROI of redesigning production lines didn't make sense.
These humanoid robots would cost $10-100k to make and would require costly maintenance for them to do the job people in poor countries do for a dollar. I don't say we won't come to it eventually, but not in near future
More realistically it seems that llms in several years could help dramatically decrease costs of automation and make it available for more industries
That's the dream.
It's easier to redesign the work to be robot-friendly than to deploy a humanoid robot and have it actually work.
ABB, EPSON, FANUC, Kawasaki... Virtually every cars in past 50-60 years have had robots and some form of AI involved in assembly. Some of them already support standard humanoid/dogbot tools in their products.
Humanoid startup execs just don't talk about that.
In the Xiaomi Model Demo, they have non-humanoid robots
https://robotics.xiaomi.com/xiaomi-robotics-1.html
And absolutely nothing in your company charter says the military won't be allowed to buy it. Y'all think we've never seen a Terminator movie before, huh.
What would be the hallucination equivalent of such robots?
I know what the post-hallucination reasoning of such robots will be:
"Dave, would you like to get the instructions on how to stitch that head back to the neck?"
falling down and breaking things?
I do think robotics would come up with more safety mechanisms (provably safe motion planning etc) just because the risk is a lot more serious than LLMs spitting half-truths
> It can better detect when humans are nearby, trigger safety tool calls and bring the robot to a safe stop if someone approaches too closely.
If the robots stop when humans are too close, wouldn't that mean that robots for close interaction or handling of humans need a whole other level of control?
Also, "better" is doing a lot of lifting here, meaning it still fails regularly.
Yes. That's a whole other level of difficulty to accomplish safely.
I always wonder why these robot forms don't have cameras and distance sensors on every single joint, processing costs I guess?
Cost, weight, durability. For cameras, bandwidth.
> we are teaching robots intelligent whole-body control, fine dexterity, *and teamwork*...
That's the most Skynet thing ever.
Wow, what a time to be alive!
I would caution against enthusiasm. Anyone opposing those who end up controlling these robots won't be alive for long.
These robots can't do combat, not effectively anyway, but that's already jumping three stages ahead, because first off, why would they?
Is the fundamental challenge for AI directed movement in 3D space the continuity of the environment? Vs LLM's which train on discrete and limited token sets.
Can someone that uses these models provide some tips or docs on how best to get to use these at home to test around? Is the best thing a virtual environment?
Do these learn?
It would be cool to have a robot that can be taught to drive the same way we might teach a teenager to drive.
No one has any idea how to achieve something like this.
Are there any AI models that can be taught?
Robots that look humanoid are a marketing gimmick.
I said the same thing about chat bots trying to act human, but evidently that's what people want. I still cringe whenever an LLM uses "I"
Its almost like Google selling Boston Dynamics was a bad idea.
Didn’t we see Chinese robots doing kung fu??
Yeah, but that doesn’t mean the west won’t try to catch up
Video has too many cuts, similar to kungfu movies where the actor doesn't know how to fight.
We're getting closer
Given the rapid advancements in AI. It is not what we thought it will, what it is, what we didn't wanted it to be, but it is great for being what it is.
What does this mean for intrinsic, the moonshot they brought back into the fold in feb?
Now they are coming for the plumbers.
Still waiting for self driving car to bring me one robotic plumber to my construction site.
You won't be waiting more than 5-10 years.
What's the point of "releasing it"? It only makes sense in the labs, and what would calling your APIs give "me" as a researcher, other than a baseline to beat? mah the model is real, but it feels 100% internal
What's the point of not releasing it? Let people play with it and get some experience with it for a price.
have you seen that NEO robot's hands, wild
* https://www.youtube.com/watch?v=QRyXV3csReA
Deepbody
we'll be having sex with robots in the near future
Sounds like a lot of work I'd probably want my robot to have set with the other robot for me
If it has intelligence of Gemini... well, good luck...
Gemini is the only LLM I've ever experienced that decided on its own to tell me that something was a bad idea. Which, ok, in itself it's not a bad thing, we actually need more of that instead of the pathological sycophancy that all current models suffer from.
But in a physical robot? Yeah, that thing is going to punch me in the face, eventually. Or worse.
I use Sonnet and Opus everyday. Believe me, it's very common they tell me that something is bad idea, write a rationale and suggest better solutions.
Gemini doesn't remember almost anything after 2-3 follow ups. I have to paste the same "system prompt" at the top of each message and it still doesn't understand it.
It's funny how Google is essentially trying to compete with the software side of Tesla only - Waymos (which AFAIK they plan to partner with major automakers), now this, etc.
That makes it sound like Waymo is copying Tesla when in fact Waymo has been in the works before Tesla even considered self driving.
Another way to look at it is that they're avoiding some of the expensive parts of making products that use AI, while leaning into their strengths. No Tesla required within the explanation.
Seems smart. There are dozens of companies that make low-margin cars. There's only one company that has managed to make a working autonomous software driver.
Tesla is the only company to have high-margin mass-market EVs.
Bizarre oversimplification.
For both manipulation and autonomous driving, google has invested in approaches with custom hardware, and off-the-shelf hardware, and a blend (which is what waymo is).