Empowering the Future

Please welcome Mark and Gil and of Course Brian heer back to the [Applause] [Music] Stage always a [Music] Pleasure all right so this was alluded To in between but the two of you have Known each other for quite some time got A photo if we could put that up right Now I asked Mark's team for a photo this Was I guess the oldest one that they Could find which surprised me because This is from your DARPA days this is Actually isn't all that long Ago 2015 something like that maybe 14 I Think maybe even before then yes yeah Yeah so Mark do you recall the first Time you met Gil well I Remember talking to Gil when uh I think I was a professor and you might have Been and were you a professor you were a Postto I think at the beginning when I Met you I was a graduate student grad St And uh I saw you for the first time yeah And and I had worked with a professor um Remind me uh his name uh who you studied With oh Jerry lein so Jerry lein was a a Real institution at uh MIT and I done One experiment with him back when I was Doing Neurophysiology which I had done when I First started grad school there and so

We talked about that and Mark you you Effectively ended up inheriting the leg Lab is that right or excuse me gil yeah So so Mark was an incredibly generous Professor and uh I had mentioned to him That I was going to tell him things uh That are going to make him turn red but Uh you know one of the wonderful things Uh about MIT at the time and about Mark In particular is that I was this this Young kid and both as a graduate student And then later as a junior Professor he Just welcomed me and so for a while both Of us were working in the leg Lab at the Same time and uh Mark had done this Extraordinary work on robots that run And uh I decided Well maybe I'll work on Robots that walk so uh that's sort of How we uh we uh had intersected then and Then when you went off to form Boston Dynamics uh you were very kind and you Just basically gave me the lab and all The stuff that was there what were the Big problems you were trying to solve in Those Days well when we first got started uh All robots that had legs were very slow Moving crab like things that would hug The ground and then you know very Tentatively take a step and hope that Things didn't topple over and then move You know like a like a spider or Something like a slow-moving Spider and uh I just took a look at that

I was interested in how animals worked I Looked at that and said wow this is About as far from right as you can get And so you know I went the other way and Said let's see if we can do something Where the Dynamics you know the energy In the in the machine was part of the Story and springs and bouncing were part Of the story and my lab worked on that So we did pogo stick robots yeah uh and You know it was kind of a leap of faith Everything was going to be harder uh but Then everything got a lot better once You were willing to take on that and I Think that's the model for for research Sometime have to go up the hill before You can go into a sweet spot where Things are going to work a lot better And I think we're there even now so Mark Goes off to form BOS of Dynamics Gil you You inherit the lab what kinds of Problems were you working on and how had It evolved since Mark say so we we Worked on robots that could walk and um It turns out that walking strangely Paradoxically is actually a little Harder in some cases than running and You can tell this yourself if you're Walking down a hill and the footing is Not so good maybe there's some leaves It's often easier to just run down the Hill than it is to walk slowly down the Hill that's why that's why our motto was You have to run before you can walk

Absolutely and and uh and Mark was right So we had worked on walking robots for a While and in particular on new kinds of Actuators that were closer to what Muscle is like so they had springs on The inside that were called series Elastic and um trying to make uh Progress on that sort of stuff we also Worked on the control and this is I Think what is leading to the kind of Work both of us are doing now which is You know how do you get a machine to be As competent at physical tasks as an Animal is even something like a cat or a Dog or even lower animals like you know Mice and it's remarkably hard I think The the the two of us you know we worked And we thought and we thought and we Thought and it's only recently that I Think that we're starting to see um you Know ways that we can really uh achieve That well you're very good at these Segways the two of you again the two of You know each other for a long time and I think what's really interesting about Your respective careers is that you've You've kind of dub tailed in an Interesting way you know you're both Running these research institutes that Are affected by uh effectively funded by Automotive Companies Gil you've been at TR for some Time in a case like that where the Company is you know again supported by

Or the research is supported by a Company that makes cars has to make Money is there a lot of pressure to Product ties so there actually is not And um you know I am not saying this you Know just just to be nice but I think One of the great things about um Companies that have existed for a long Time is that this isn't the first time That they've done R&D and certainly most Of the R&D budget inside of Toyota is Spent on figuring out how to make the Next cost car maybe the car 5 years from Now but there's this U notion and I'm Sure that Hyundai shares this as well That we're at the once in a century time Of transformation for the car industry And Battery electric vehicles the new Kinds of cars that are being made uh are Much simpler to make than the cars have Been in the past no engine no Transmission Etc and so we're going to Have to compete in a much fiercer way on Cars but also can we use the skills uh And the dreams and the hopes of the Company to go beyond cars and so tr's Job is actually think about what's next What's going to be next after cars as Well as some kind of fancy stuff on cars And I would I would if I could say for For Hyundai I think uh in fact I've Heard Gil say that making a car is Taking small bits of metal and banging Them up and then welding them together

Or attaching them somehow okay and you Know that's a little on the Primitive Side and the and you know now we have Software robotics AI uh all that stuff and the car Companies need to embrace that and I Think that the leadership at Hyundai you Know the chairman visited me last week Uh we're in you know close touch with Him and the people who work for him and They think that getting into the 22nd Century is an important thing to do and Uh you know we're getting started on it Now and in in addition just to build on That there's there's actually this Al Ignment that's happening between uh cars And Robotics as well I think actually Rod Brooks was the one a few years ago Who said that you know modern cars what Are they they're Elder Care robots and I Think that that's really true you think About the amount of computers in them The amount of software that's in in them And so actually TR has different Divisions inside of it uh one of which Actually works on very Advanced things In cars that is completely overlapping In some of the software some some of the Concepts that we're using in the robotic Space Elder Care is just a big Focus for TR in general and I think a big Motivating factor is the Aging Population in Japan how is it essential to the work

You do so we think about this problem of Aging society and it's actually kind of A worldwide thing um it's not just Japan It's going to happen in the US as well As a result of the baby boom after World War II but not quite as bad uh but it's Happening in many different parts of the World and the truth is and I'm sure this Is true for all of you um you know Growing old isn't Fun and uh Particularly for our parents and our Grandparents you know we've all watched How difficult that can be and so the Question we ask ourselves is you know Can't we do better can't we improve Quality of life and of course there's The physical part of it that it's sort Of you know hard to bend down and grab Stuff as uh as uh we did when we were Younger uh but we're also looking at the Social social side the loss of purpose Uh the um sort of feeling of what's the Meaning why am I here do I still have Something to give we joke all the time That our goal is to build a time machine What we really want to do is take 20 Years off of people's lives uh I watch Futurama just like you know lots of Other folks I don't know how to build a Time machine but maybe the machines that We do build uh could have the capability To allow people who use them to feel That they can still do what they could Do when they were younger that's the

Goal Mark you you and I have spoken About this a few times but you know you Were leading Boston Dynamics for a long Time sounds like you weren't entirely Sure what was going to come next Once Boston Dynamics really started Productizing what was your early pitch To Hyundai for this large amount of Money that they gave you uh the pitch Was very simple it was that working on Products was only going to take you so Far and you really if you're going to Solve the big problems in robotics you Had to have a longer timeline and build Up a team uh you you want the kind of Teamwork that you get a corporate lab But you want the kind of futuristic Thinking and uh not caring about Legacy That you get at universities and that You could have a lab that was a mixture Of those two and uh that was the concept For uh what we do at uh the AI Institute And uh it was very short pitch uh I Think eight pages and they went for it You contrasted it with corporate R&D Divisions but how is the work that each Of you do different than obviously you Both have University backgrounds Gil was Doing that for quite some time in this Kind of Institute what's the difference Between that and the university yeah uh I think that you know this isn't true of Every University but mostly it's Professors working with a team of

Students the students have some skill SKS but as soon as they really have Skills they they disappear they go get a Job and uh at a place like Boston Dynamics and what what I hope will be The situation at the AI Institute you Have teamwork where uh people have Skills at all the various disciplines They're brought to bear on a project you Can have uh a larger scale of activity And you know this session is called Hardware but having uh being able to Work on real hardare W development and Have it mated with real software Development and AI uh it's really hard To do that in a university scale Lab and I would also build on that at TR It's very much the same sort of thing There's also a thing that I learned at DARPA that I think both of our Labs Share uh which is this thing called Pastor's quadrant and the idea there is That it's use inspired fundamental Research and many people get confused by That because it's not on a single scale Uh but it's this two-dimensional idea And we're always thinking about you know If we're successful how can this Suddenly enable all kinds of new Products but we're not driven to solve a Particular problem with a a product the Way that many of the other R&D efforts Are at the same time we want fundamental Understanding so we're acting very much

Like academics like scientists asking Why does this work and how can this be a Transformational change and I would go Out on the limb and say that We so I've been giving talks in which I Say that we want to uh be the Bell Labs Of AI and Robotics but we also want to Have no products because as soon as you Have products you have customers and Then customers have things that they Want like 10 features that you didn't Have so if you have a 100 customers you Have a thousand features and those Features might not be in the direction Of the more significant solving the more Significant problems that really prevent Robots from being what we dream about You know as smart as people able as Agile as people or smarter and more Agile one thing that really jumped out At me when you made the initial AI Institute announcement is that it's the AI Institute you know you're you're a Hardware guy it's it's not the robotics Institute obviously some crossover but Why is that the main Focus so first of all we are equally Focused on Hardware okay and and AI or Cognitive intelligence the name the name Is was a a side thing AI was hot it Really should be the AI and Robotics Institute robotics institute's already Taken so there there was just some Politics we are actually going to

Rebrand ourselves uh soon you know our Official name is the Boston Dynamics AI Institute which causes all kinds of Problems because we're a totally Separate company except for that I'm Involved in both uh and so everybody's Confused and think thinks we're you know Connected the hip which we're really not So we're going to we're going to work on That okay you haven't picked a name yet No okay but but AI is something that You've really been focusing on recently Yeah that whole idea is to make the Robots physically able like the Boston Dynamics robots are but to make them Smarter so you can interact with them uh You don't need a fleet of programmers to Program every skill like gills group has Been showing and I guess you had a press Release two days ago and and did you Show stuff here uh we haven't yet okay You're you kind of showed my hand a Little bit there Mark well that's actually that's a good That's a good segue if we could show the Video right now so as Mark alluded to This news broke earlier this week I Visited you in your labs and uh wrote a Piece about it Gil talk us through what We're looking at right now sure so this Is actually some older video from uh Last year and what it's showing is Things that we achieved in a very narrow Slice of the robotics work that we do uh

That's actually done by programming and You can see some of the code in the Background here and that's a before shot This is an after shot this is the more Recent work that we're doing and as we Discussed in the press release and we're Going to talk about even more uh in this Case we have figured out how to do Something uh which is to use modern Generative AI techniques to enable human Demonstration of both position and force Uh to essentially teach a robot from Just a handful of examples uh the code Is not changed at all between any of These examples and what this is based on Is something called diffusion policy It's work that we did in collaboration With Columbia University and MIT we've Taught 60 different skills so far and The goal is actually something that you See here called Fleet learning which is When one robot learns something all Robots learn the same thing and that They can comp pose those skills together Uh in a single Hole uh there's many many Steps that we're going to need until we Get to robots that are as capable as People or as animals but we think this Is a very significant step that we've Made and a bit of a Breakthrough can you draw a parallel for Me between large language models and Large behavioral models yes so there Have been um other groups that have been

Doing very good work where they take a Large language model and essentially use The robot as an input output device so The sensors of the robot and maybe a Little speech recognition is used to Translate what the robot sees and what Is said into a language command to a Large language model that runs just like Chat GPT runs it spits out something to Do and then the robot kind of in the Output mode goes ahead and does that That's not what we're doing uh you can Think about that like two things made it Together like this the robot and the llm What we're doing is actually putting Things together where uh the diffusion Policy what we're using is actually Integral to how the robot itself learns Each one of the skills and how the robot Executes the skills and I think that's What's so exciting about it and I'm sure There's going to be lots of work from All over the world on this kind of way Of doing things so instead of using Language instead of using words we're Using both words and also input from Human beings and also sensors and Actuators from the RO itself so we're Changing it from large language model to Large behavior model one of the things That you told me that was interesting is The contrast between this and what bosta Dynamics has been doing traditionally Obviously you've got a lot of robots out

In the world they're out there Interacting with the Environment how distinct Mark is a is an Approach like this from what you've done Traditionally how what how distinct are The two approaches uh traditionally yeah They're they traditionally we program The robots by people creating Control Systems uh lately those are uh MPC or Model predictive control systems and boy That stuff has really gotten Sophisticated including very recently I You know I kind of find it painful that People call that the traditional method Compared to learning method but it's Very powerful I think it's going to have A continue to have a life and and a role Uh as things move forward uh but that's Different but the you still need uh room Full of very skilled uh and accomplished Programmers who spend a lot of time Making you know the kind of things You've seen in the videos of Atlas Running around the parkour course or uh Throwing the bag up to the guy on the Higher level if you watch YouTube you've Seen those things um this is really a Different Paradigm you know Paradigm That we're work hoping to work on or Accomplish something too we're only a Year old so we don't have anything to Show like uh like these guys but the Whole idea that you can tell a robot to Watch someone do a task and then be able

To figure out how to do it can we show That uh that picture yeah let's let's Pull up uh the first slide I think it's On Deck There you know the the model here is uh Rather than have a room full of Programmers tell this robot watch that Guy un this is called watch understand Do understand what you're seeing and Then do it yourself and there's there's A lot of richness in that problem you Know what does understand me Uh segmenting the various tasks having a Concept of uh of a sequence and what What's trying to be accomplished Figuring out what skills to bring to Bear if you already have some available Or learning new skills from uh from Watching the task being done it's very Rich problem uh and we have a a team Working on that and starting to make Progress the large language models play A role but the the tokens in a large Language model which are words aren't Grounded in the physicality of tasks Like this which involve uh joints and Motions and energy and forces and Compliance when you say open a door to a Robot it doesn't necessarily know what That is or how to execute it right it Might not know that it it might be able To figure out from a language model that Has to grab the handle and turn it but What about the traction that your

Fingers need on the door knob in order To turn it and uh is it a push door or Pull door and where do you have to stand And if you're a quadruped walking Through a door it's not the same as Being a human or a human or a humanoid Going through the door so this you know There's a lot to do uh to go from a Simple description to doing it on the Other hand if I asked either of you to Do those tasks you'd have absolutely no Trouble you know well I might I might in My current state but yeah generally That's true there is a there is a a part To this that's also I think very Important to point out that is new there Have been um successful uh research Attempts before to get a robot to mimic Uh what a person does for instance if You uh do a Telly operation like you see At the beginning of the movie that we Had played um getting that to play back And having the robot duplicate what the Person does perhaps with a little bit of Robustness if things aren't quite placed The right way that has been done the new Thing what's so exciting about this is This idea of composition so for instance Um if any of you are asked to do a task That you've never done before well the Fact that you've lived from being a Little kid up until now uh actually Provides you with tremendous numbers of Skills of competence that you can

Compose together and blend together in a Way to do the novel task that has not Been possible before and so what's Exciting about this is it's not just That we're learning from people but We're learning in a way that things can Compose and if you think about large Language models that's what's so Exciting about about them as well that They can compose from many examples in Language from what they learn to come up With something that's a blend that makes Sense we're hoping that exactly the same Thing is going to happen now uh with These behavioral models one of the Themes along that line is we're Interested in Mobile manipulation and Taking advantage of a of a robot that Moves around and you know if you guys Could all do an experiment which is to Sit at the dinner table tonight and keep Your shoulder stationary and see what You can reach on the dinner table it's Approximately nothing you know maybe the Plate but you probably can't reach your Water whereas if you just have the Mobility of doing this you can reach Everything that's in front of you and of Course if you can get up and move around And if you're technical about the Details of how an arm works you can Relieve singularities by having Mobility So there's a a whole other world and Really that's the composition of

Mobility with with uh you know arm or Hand based dexterity which is just an Example of what you're talking talking About yeah let's if we could Advance Through the slides um one of the things We're trying to do we we we're not Actually building applications but we're Trying to keep in Focus both industrial Uses for what we're doing and consumer Or uh home type uses so this is another Watch understand do situation where you Watch someone cook something you figure Out where to what's going on and and uh You can do it and we also have a project That's more focused the next slide uh on Uh Mobility you know very Advanced Dynamic Mobility I tried to sell this at DARPA about 10 years ago the idea of Putting wheels on the end of legs and Throwing Mass around and uh uh they said It wasn't far enough out to be honest That that that this is too realistic What we're looking at right now uh that The idea of having wheels on legs was Not uh aggressive enough in uh in terms Of pushing the boundaries which I don't Think it's true but that's what the at One time was the reaction I got I I have To ask you know obviously you're the leg Guy you have been the leg guy um there's A big debate you know legs versus Wheels What do you get from putting wheels on Legs oh well you get all the opportunity Of being able to go fast if you have a a

Prepared surface uh but you can also go Over anything uh I'm not the the leg guy Anymore that was that was before now I'm The everything guy Well and and I would point out that you Know there's a lot of renewed interest In humanoid uh robots as well I know um That you're going to have some here uh From our good friends uh at agility um You know it's incredibly neat and Wonderful at the same time I think that We have to always keep in mind that There's a human tendency to project a Whole bunch of things on a robot based On how evocative it looks and the reason Is that you know we have this evolved Thing in our brains where we recognize Other people and it creates certain Emotions inside of us even when we see An an animal that moves in a certain way Um you know there used to be folks who Wrote in response to the YouTube of Videos that were at Boston Dynamics all Kinds of emotional things that they Ascribed to the robot that was being Pushed a little bit and oh that poor Thing you know I'm going to correct you The used to that is very much still Going on yeah so why does that occur and It's because you know we we have this Kind of Eliza effect where we we just Identify with the machine when making Engineering decisions what the form Should be there's a lot of uses for

Humanoid robots but it's not a cure all That's only part of the answer and the Hardest answer is what's going on in the Robot's brain and also in the network of All robots together that are helping Each other to learn I I want to add to That uh I don't know if how many of you Remember uh the expression if it walks Like a duck and it talks like a duck and It I don't remember what the next one is Then it is a duck but in robots that's Just not true it can have two legs and Two arms like a human it can climb Stairs like that where it looks like a Human but it doesn't have the emotions Of a human or the morality of a human or The necessarily the intelligence or the Language and you know robots are these Constructions where you can decide what Features and elements are going to be in There and then you know work on on those And they're they're not put together the Way humans are just because they might Look like a a human in some respects and That's a real confusion to the public at At large I think it's interesting as as We were looking at the cooking slide you Know it occurred to me that that's not Dissimilar skill from what your team has Been working on um and in that case and In the case you know I visited your Labs A while ago it is effectively a humanoid Robot or at least a robot with a Face why is that important for a robot

Like that so there are elements of the Built environment U that we all have and The robot has to have a form that Somehow Works in that environment uh the Humanoid form might be the one that you First think of but it's not necessarily The only one so for example a while ago Uh we had a machine that was a robot arm That was hanging from a Gantry on a uh Ceiling as an example um so again it's Important not to get too enamored with The form both of us uh love the hardware Side of things but uh and there's still Work to be done there and it's exciting And it's great but there's also all this Other work on the intelligence in the Machine itself and even more importantly What's the purpose of the robot even if You're not worried about products right Now uh our view is that like I told you Uh we want to effectively help people be Younger and so what that means is Helping them um amplifying them but not Replacing them because if you replace a Person if you have a robot that cooks Meals for you while you sit and do Nothing you're going to feel worse You're going to feel that you have no Purpose on the other hand a machine that Helps you as you cook in order to help Make meals for your kids and your Grandkids and your friends you feel Younger because I can still do something Meaning

So let's let's throw up the final slide This has uh a lot of the images uh Mark When I visited your offices in Cambridge If you walk into the lobby these are all Up for everyone to See why create renders like this and you Know what role does this play in the Actual work that you Do well we have a a list of projects That are related to these pictures and Others that that are in here we're not Working on all of them yet yet but it's It's an idea for what to work on um you To be honest in these early days I used To go around and only show videos of Results and that's the way I'd give a Talk well we don't have any results so This was a way of uh kind of conveying The kinds of things we were we're hoping To accomplish uh so you know I just Worked with an artist these are not made By Dolly which a lot of people ask me About these were made by uh you know me Talking to an Artist uh and then he C something up and I would tweak it a little tweak uh the Instructions a little bit and he'd make It a little different and they're just Meant to convey projects we want to work On work on just walk sorry watch Understand do uh one of them is called Inspect diagnose fix uh Dynamic mobile Manipulation uh and a bunch of others we Only got a few seconds so I'm going to

Ask you both a really big question and Give me a shorten answer as possible how Far out are we from general purpose Robots I want to distinguish it from um human Level AI because I don't think we need Human level AI for general purpose uh Robots I think the answer is between 5 And 10 years from now and the reason I Say that is that it has to be true Because if you look at the Aging Society Curves that's when we're going to need Them it's going to take a while for them To spread and for the businesses that Make them to grow so our own timeline That we're using which is set by those Aging Society curves is 5 to 10 years From now you know I I generally agree I Think you could argue that even spot the Spot robot you know we have about 1,200 Spot robots out there and if you look at All the different things people use them For there's some argument that we're Starting to embrace it has an SDK which Helps it has an SDK it has a a hardware Roof rack so you can attach whatever Stuff and there's you know there's a Connectors and things and so people are Adapting it for all kinds of stuff now You know the intell Ence you need to Make all that work is still a work in Progress but there's certainly a lot of Generality to the things that can be Used on the other hand I think the the

Sense of your question is when are they Going to be more like uh like people and Destructible and I think youil you know Gil's right maybe the outside of that More like 10 years I mean it took Tesla 12 years to go from their first demo Electric car to the model S yeah uh and You could argue that that was a pretty Straight path with a pretty well-defined Market uh and we have a you know a Tougher challenge than that for robotics Great we are all out of time Gil mark Thank you so much thank you so much Thank [Applause] [Music] You