Lights, Camera, Algorithm! Where AI Fits into Movies, Games and Creativity

We have ritika Kalia from Runway Ty Sheridan from Wonder Dynamics and John Spitzer from Nvidia please welcome them As well as our moderator Devin called Away to the stage please let's give them A hand [Music] [Music] Well well back again uh thank you very Much for joining me Um in lieu of asking you all to Laboriously explain what it is your Companies do although I'm sure many of These people will have heard of them I Ask you to supply some some video and I Got some John you didn't send any over But I got it so we're just going to run Really quick a video I think here That'll show off there should be no Sound but we might see it down here Right here is Wonder Dynamics with some Uh Drop In Motion capture and production Stuff We'll see that going We decided with no sound but I love that And here's Runway this was what you sent Over I love the snail all generated Imagery And then here's an example of just of Some generative speech uh being done in In an engine in game uh by nvidia's uh What is the name of this Tool uh this is Called Kairos it's the demo that for uh AI powered NPCs yeah

Uh so I just much much quicker to show You than have us all discuss that but Thank you very much for joining us Um and also as a preliminary I thought That uh considering this topic we should Probably acknowledge the uh the writer Strike and the SAG strike and everything Else that's going on in Hollywood since A lot of this Bears directly on that and to that end I Thought Ty maybe you could kind of uh Talk us into this since you sit on both Sides of this this issue sure maybe just Sort of what's what's going on here yeah So I'm an actor by day and entrepreneur By night Um and uh yeah so at Wonder Dynamics you Just saw a demo we essentially Um you can almost think about it as an AI solve uh for motion capture but not Only motion capture but lighting and Compositing and basically a large Process that takes a lot of artists and A lot of time and it costs a lot of Money so for us really the focus of what We're trying to do at Wonder Dynamics is Speed that process up and not only to Speed it up and help production save Money but also give people more access To these effects Um but yeah I've worked on you know as An actor of small films and big films And um and I think you know I've uh in 2016 I worked on this movie called

Player one we spent six weeks in motion Capture and the technology there is Really really cumbersome so I think a Lot of those films don't necessarily get Made and you know there's a handful of People within the industry that can get The budgets to go and make those films And so I think for me the big excitement Around AI is is democratization you know An expansion of the market and you know Enabling storytellers to have their Voices heard maybe for the first time And without those tools you know maybe They wouldn't be and I think you know It's it's uh Hollywood I think is Notoriously kind of standoffish to new Technology which is which is uh which is Funny because you know we wouldn't have Cinema without the Advent of a camera so I think technology has really been Um you know this the single element Pushing Cinema forward and allow us Allowing us to do newer and newer things And sell new kinds of stories and and I Think that's really no different of Course with AI you know we have to think About using it in the most responsible Way and and Um I I think that you know the using it To empower artists to keep them a part Of the process is is a deep kind of Philosophical Focus for us at the Company and I think that's kind of the Tune I hear of the people seeing in the

Industry so I think as long as we're Doing that it can be a really really Exciting time I think that people tend To simplify it a little bit and you know To the maybe tune that that AI means Taking people's jobs but I don't I don't I don't think uh I think it's more Complex than that and I think you have To consider how it's being used in each Business yeah I think that you're right That it the the dialogue is Oversimplified and I feel that there's This but there is this sense there's Still this sort of vibe that AI is sort Of creeping in at the edges or that it's Invading in some ways uh so I wanted to Ask John uh for instance you've been You've been working with this technology And with AI AJ's and stuff and all kinds Of stuff at Nvidia for quite a while Like you've seen this kind of thing come And go can you maybe speak to you know Is this new or has it been around He actually started working on AI in 1984. there you go so it was um I was Doing a digital twin of a uh it was Actually a chemical plant not far from Where I lived in Houston so I'm from Texas City Texas which sounds really Cool unless you actually know very Texas City is because it's a Refinery town so It's not really some place that you're Proud of being from but um yeah so I I Created a digital twin as a uh as a

Science fair project for my senior year And it was driven by an AI which was Expert system based then not neural net If I'd known now what I know I knew then I mean vice versa I would I would use Neural Nets but um you know I've been Working on AI for for decades now and I've been working on 3D for the last uh 30 years so um I think it's super Exciting what's available uh in terms of Applying AI to both runtime Graphics as Well as to accelerating game development Just like Tai was you know referring to Democratizing the technology because Right now it's a very steep learning Curve for many of these tools for Creating content for games or for films And uh to to be able to open that up to Independent developers who don't have 100 million dollar budgets to go and Throw at a project I mean I looked at The top games that were released in this Past year and 95 of them were sequels Now it's nothing wrong with sequels Right I mean you know we all love you Know a certain IP we want to see more And more of that IP but and the one that Wasn't a sequel was was Harry Potter you Know so obviously a very well-known uh IP with with many books behind it so um I would love to be able to Give independent game developers the Same tools and the same ability to Create incredibly immersive and

Compelling content and I think AI is Going to bring that to fruition And ridiculous you were coming from a Like Runway is sort of coming out of Nowhere like really blasting out with This like uh generated video which is so Fresh and new that I like I don't know Where to even start talking about it Like but you have actually experienced Not just building it but seeing how People are using it does this tally with Sort of your your feeling of how people Are using it it's democratizing it's Enabling is it is it doing something new Or is it I don't know how what is your Experience yeah for sure I think that's A big part of what makes it really Exciting because we build these tools And once they're in the hands of artists And creators we see so many ingenious Creative ways that they're using it Incorporating it to really also tell new Stories to get more diverse voices to be Able to tell the stories because you Don't need the most expensive computer Or approvals from the right people to Get started like with runways Gen 2 we Have text to video image to video that Barrier to create is so much lower that You can go from an idea get something Out there get feedback start building And you know we're at the point where We've had short films by Paul trilio Thank you for not answering that was

Generated just with Gen 2 all the frames And this was a very moving film to me Personally as well you know at explores Themes of loneliness but it also got Rave reviews from The New Yorker we've Also seen it being used in music videos That combines footage real footage with Like style transfers using gen 1 Personally for me my mom's a teacher When I showed her gentle she said I Would love to have my students explore This as we're reading a book together And we read a scene and people have Different interpretations of what Happened they can not only express Themselves but also have a discussion About different versions of what Happened and so I think the diversity in The use case almost in any uh workflow Is part of what makes it exciting and Really empowering the creators and Making it more accessible to tell your Story I I feel I agree in a way but also In another way it seems almost equally Obvious like automating motion tracking Generating conversations uh generating Imagery wholesale what are these Replacing if not animators writers Photographers painters I know that they can be additive but are They also replacing people are they also Obsoleting people yeah like for for us Our goal is really to make tools for Artists and there's some examples I

Think sometimes Um you can look at a task like Rotoscoping that can be very tedious Take creators hours and one example is In the movie everything everywhere all At once the rock scene use Runway to uh You know kind of remove some of the Shadows with rotoscoping you can Actually remove that that tedious Process and something that would take Hours can take minutes and that's giving Creators time to try out more ideas So I think there's also that that angle Of like being able to have that time and The cost to experiment is lower now Overall with AI I think a lot of our Jobs will change but there's also a Difference between a job and a task so Some of the tasks that are in our jobs May uh be automated or will evolve but The job of a Creator that's so Central To telling stories I think uh that's That's gonna remain Parts major change As will all of ours but Um gotcha yeah and Ty you mentioned Earlier like the the laborious process I Was going to ask about when you're Filming Ready Player one very heavy on Virtual production on uh virtual Characters representations and motion Capture uh Has that has that changed or was that Really like sort of laborious enough That you were like all right I'm out I'm

Going to start a company to completely Redo this whole industry it wasn't Really as laborious for me as it was for Some other people I will say that Um but no I think that the exciting part Is that you know it's you see these Fantastical movies and there's so much Fun to watch and it's an amazing Adventure and experience and a spectacle And then you start thinking about how Much they cost how difficult they are to Make how many people it takes to make Them and you realize wow this is super Limited right there's only very few People that can make these films or even For actors you know it's it's it's you May have less opportunities now than you Will have in the future when we have More tools that are enabling us to Create these effects so I think that's That's super exciting right Um but yeah it's it's beyond just the Motion capture you know the way and and You know this very well but you know 3D Uh so for instance if you open up unreal Um you know sometimes I think for a Beginner it can be really intimidating You know don't know what to do in these 3D softwares they're really complex they Take years to learn and if you are a Master at unreal it does not mean you're Going to be a master in Maya you know so Each of these softwares are a career Into itself right and so I think for

Someone that's a filmmaker or Storyteller and they just want to get Their story told they like you said you Don't know where to where to start Um and so I think even to conceptualize Stories using AI tools to conceptualize Stories is something that's incredibly Exciting as well Yeah I think a lot of the Film Production game production it's tedious Work and you know I I there's I think They're parallel in our run time with uh Our gaming sdks as well we have a Product called dlss which Um pretty much speeds up everything While keeping the image quality the same And it's doing that using AI but you're Able to then free up a lot of the budget On the GPU you know the graphics Processing unit to do more interesting Work because actually only one out of Eight of the pixels that's displayed to The gamer is actually rendered the other Are all inferred using AI yes so the Interesting work right that allows you To have that much more time almost seven Times as much time to do interesting Work to create more beautiful images and To have more fluid motion and animation Sequences and textures and all of these Things which are innately human the Creative art is not going to be replaced By a computer It's the drudgery it's the TDM that are

Going to be taken away so that people Can be more creative absolutely and Certainly I've certainly heard that from Other game developers uh Ubisoft was an Early one who was trying to uh not fully Proceduralize but uh increase the the Power the output of Any Given developer So that you're not sticking one tree Down two trees down three trees down You're like I'd like a forest and then You say what kind of forest a dark Forest okay now now we're getting Started Um and you mentioned dlss I'm curious Actually Is has there been any interest in using Dlss for like AI upscaling restoration Within TV and film like is that an area That you're also getting into because it So clearly has these these applications Within gaming but does Netflix call you Up and say like hey we'd love to have Our bandwidth usage I don't think I'm Allowed to talk about any of those sorts Of deals Even if I knew I knew about them but um Yeah so certainly for previewing I mean You know I think one of the big things In Film Production I'm not an expert Here like Ty is I mean but a lot of People Pre-visualization right I mean so you Know getting the right shot you know and So having that you need interactive

Frame rates so that and they're using Tablets with ar to try to go and capture The right film motion or the camera Motion and all of those sorts of things And absolutely dlss is immediately Applicable to any sort of real-time Applications during the production Process now of course in the actual Rendering of the final frames for the Film no I mean that doesn't make any Sense because they're spending you know Like you know thousands if not millions Of cycles per pixel you know and so Every pixel needs to be perfect in that Case and dlss would not be the right Tool for that Uh I in in the production side of things It's it's my experience my limited Experience that uh at the pro Enterprise Level or uh many many studios and things Like that they don't want to use a tool Unless they can be 100 sure of the Output and the ownership of the output Is that something you run into in your As you're trying to engage with sort of Higher level creators or groups of Creators Yeah I I think that's a question on a Lot of people's minds as they're Exploring the space Um we've also seen an openness to try Out different AI tools also like I Mentioned the rotoscoping examples and And what that means I think there's a

Spectrum of tools that are available Like we have 30 different tools and Um we kind of think of it as we're Building a new kind of camera and if you Look at it from that lens when the first When camera first came out there was a Lot of questions around what does this Mean for privacy or what does this mean For other mediums and now we can see we Both have you know regulations and Social norms around that and the immense Benefit that we've all had entire Industries that were created from that So where we are in that moment now is Like that really early stage of this new Camera and we're trying to figure out How do we work with artists industry Partners to best navigate this so that It has the most benefit for everyone Including artists so that's kind of the Lens with which we look at questions Like that I suppose as as you kind of this is a Good as good a place as any to talk About artists rights and ownership uh I Think maybe Ty you can tell us this is It's kind of different for actors I I Feel like part of the fear is of uh Giving away your likeness uh you know in A virtual production or something like That you're not 100 sure like what did The contract say about reusing my face That kind of thing I know that there's a Little bit of fud around that but you

Know is that a reasonable fear and like Going forward the worrying about like Essentially I mean it's it sounds crazy But like having an AI driven like Version of yourself instead of an actor It's like five years ago I would have Said that's the dumbest question asked At this conference but now I'm like if I Don't ask it now I'm going to be kicking myself in a year Or two yeah yeah well I mean this is It's the world that we're living in you Know and I think we as a community as an Industry I think we have to be raising This question and having a conversation Around it both you know on uh one side Of the strike and the other you know I Think the whole you talked about Um people who work in Ai and I think People who are creatives or artists you Know we need to come together and be Having this conversation I think there's Some really exciting use cases for Instance you know if you want to age Someone by 50 years suddenly using that Person's likeness become very Interesting right if you're using Someone's likeness obviously against Their consent Um that's obviously unethical uh so I Think it's it's it's it's it's it's very Interesting again it's like going back To the complexity of this it's a really Complex

Thing but I think ultimately an artist Should be able to consent how to to how Their work is used or their likeness and And and I think that we really have to As an industry come around artists and Make sure that we're supporting them in The most ethical way as possible and Using AI tools really to contribute to Their experience right as only as much As they support them and contribute to Them in in all the ways that that they Want I think that's um where we we have To try to land And what about uh for for Runway like That I mean that this is an open Question for uh companies that are Putting out commercial products or Products that are uh maybe trained on Copyrighted material or some artist Material or something uh just today uh Nvidia video sorry open AI announced uh Dolly three which in the you know small Print they're like oh it will try to Refuse to draw anything in the style of A living artist if it understands that That is the case it's an interesting Step to take uh but of course it also Acknowledges that Dolly three was almost Certainly trained on the work of living Artists which I don't know how you could Make an art model that wasn't but how do You navigate this this like it feels Like you're just navigating a Minefield There's no way to win especially in this

Question that I'm asking you like yeah Um I I think the what's important to me In to me is also like our Founders at Runway are artists themselves as are Many people on the team so overall the Way we navigate this is with a deep Empathy and centering around the artists And the creators and kind of like what I Said if this is a new kind of camera This is really early and the space is Evolving so fast so what's important for Us is to be really plugged into those Conversations that are happening and Really learn and listen Um as this space navigates and and Continue planning for the future with With that in mind and what about Transparency on the data sets and things As you're moving you know because you're Constantly training new models you're Doing new things you're taking new Approaches you know is there is there Any is anybody advocating for the Artists that are you know being included Or excluded if you if they ask if they Don't ask there's I mean the the sort of Contours of consent and consumption here Are extremely complicated as you've said It is it's very difficult to navigate But Like is there is there a possibility That in the future you may have to have A model that is trained completely on Non-copyrighted work or non-living

Artists or pre-generated art or licensed Art how does how is there a way to do it Ethically And I think you're starting to get into Like different segments of what models Can be and what's out there now versus What the space could evolve into and Um although I can't share specifics About our roadmap but like I said the Way we look at it is like we are also a Team of artists and we're working very Closely with them um and with industry Partners to to navigate this and see What kind of segments come up and and if There's ways to uh different ways to Incorporate that into how how we build For the for the future yeah the approach We're taking at Nvidia is to partner With uh large uh well people who have Large amounts of data but that have been Ethically sourced legally sourced with Consent from the owners of the IP so Like Getty Images or Shutterstock and so Creating foundational models based upon Those data sets uh and then of course You know there's an agreement that we Have with them so that they're properly Compensated for it but then you could Then partner with a large company let's Say we have have a game publisher which Has you know 30 years of Ip though for Multiple franchises and so they would Then be able to use our infrastructure To create a fine-tuned model of that

Foundational model so they have all of Their data which is then added to all of Let's say Getty's data and you could Have a text to image or an image to Image generator which was completely Ethically sourced Now I think I think the The Hope here is That there would be a technical solution For this but I mean every time I talk to The researchers that is about getting Attribution to the the individual images That that actually contributed to the The resulting image I'm talking Specifically about text damage but it Could be text to video Could Be You Know Audio it could be music right but you Know then you could say oh I want to Have this sort of mashup of you know Various IPS and then it would dull out The appropriate micro transaction for Each of those IP owners but I don't Think that's possible quite yet people Have taken the approach of either having A deny list on the front end of The Prompt and like you said you know uh Denying that to ever actually be sent to The AI service or on the other side Where you're taking the uh the output And then applying a classifier to it and Then trying to figure out you know what IP went into that image but you know That would be easy if it were a known Um you know like Harry Potter you know We know who owns Harry Potter but let's

Say it was the Eiffel Tower you know Maybe in your data set you had a Thousand different contributors who all Had the Eiffel Tower do you give a Thousandth of a cent to each one I don't Know yeah so it's clearly it's clearly An unsolved problem and it's like that We're all you all have different sort of Takes on this uh I think it's uh it's Interesting that Ty you you sort of Decided to work outside of the uh the Structure that you were already in the Filmmaking industry where you were like Well this was very difficult uh and you You mentioned when I was talking with You and Nicola the co-founder uh That you you wanted to make films but You couldn't because there was there Wasn't the the budget there wasn't the VFX availability that wasn't this and That and uh I think I just think it's Interesting that instead of going and Buttering up a few producers and trying To you know Grease the wheels and make It work from inside the system you're Like no I'm gonna start a startup and Like a deep Tech startup and then spend A couple of years you know baking that How did how did you make that decision Like it just seems like an unusual Decision it was not that black and white Tell me about this black and white Decision spur of the moment I woke up One day and yeah

Um no I think we've always been very Curious guys you know we are super Curious about new ways of telling Stories and I think You know yeah we my co-founder and I Originally Met 10 years ago working on a Film and we were both writing our own Projects at the time and we kind of Bonded over that and started sharing Material back and forth and then we Started developing our own projects Together And then as we were developing we were Like oh man this is going to cost 150 Million dollars who's going to give us 150 million dollars to pay for this Surely they're not going to give it to Us and uh yeah so we knew it was an Issue and we were like well it's a Feasible way of going forward how could We make these films and we want to make Tell these stories and so we started Dabbling in technology and uh really it Took on different iterations uh over a Long period of time I think we started Building first dabbling in technology in 2015 2016. so um we kind of pivoted more Towards AI in 2018 when we were Researching Looking at research papers from the Field of Robotics and autonomy and all These image perception models and real Realized Holy crap this is gonna inevitably be

Our industry in one day you know this is Going to change how visual effects are Done for sure And because film is just 24 images a Second right and uh so we kind of just Started Building Technology selfishly Because we wanted to use it to make our Films and then we realized this is much Bigger than us you know there's a lot of People out there like us that have Dreams they dream of making these movies Or telling their stories and they don't Have a way of telling them what if we Turn it into a platform and make it Available for everyone and we can Continue to utilize the tools that we Build and and tell our stories one day So that was really the evolution of of The company and right now we you know we Always say we're we're just as much a Story company as we are a technology Company we've we've gone through the First phase of building a foundational Technology we're now entering the second Phase where we will start to develop our Own stories that the company utilizing The technology I I you you mentioned earlier actually Uh when we were talking before that one Of your first projects you tried was uh Procedurally generated dialogue uh or You were trying to do like uh yeah Virtual characters yeah and now John Like what we just showed was actually

Sort of an execution of that so I'm kind Of curious like uh why you didn't pursue It and now John yeah have you have you Succeeded you know had the Mad Scientists like finally it was the tip Of the iceberg I mean we're we're just Starting to explore uh what what this Whole thing is capable of doing I mean You know I think the speech recognition We're good there right I think the uh The Texas speech there are some very Good packages out there for Texas speech But when it comes to the uh the large Language model you know most of the work Has been done on chat Bots and you never Really want a chat bot to lie to you or Try to deceive you or try to put you on The wrong track whereas non-player Characters in games this is like sort of Like standard fare you don't want you Want them to lie to you you know and so Having that you know that ability Ability to imbue certain characteristics Into a character and it has to be a Broad range of characteristics right so Having that while maintaining some Control over how the character is going To converse with the player Um and you know you see all these Articles about people like jailbreaking You know chat GPT and doing all sorts of Crazy stuff and if you're a large game Publisher you can't have that happen you Know you got kids playing the game it

Needs to be safe so I think that's going To be the largest hurdle for us to cross Is ensure that we're keeping these Non-tech player characters on track that We're keeping them you know on the the Role that they're supposed to play Within the game like reveal this but Don't like try to do trigonometry right You know you don't know hyperbolic Functions but you do know like how to Create a knife or a an ax and then by All means you know don't be Inappropriate at any time so I think That's going to be a tough tough thing For us to solve but we're up for it I Mean it's definitely going to be a great Experience it's trying to figure this Out and amazing it's amazing how all These conversations some of the most Complex problems come back to ethics you Know that's right fascinating well I Want to actually uh I feel like Nvidia has already scaled is My feeling but I want to talk about Scaling with you too would you you have An idea it goes from a research project To an actual like a business something That you're like well if I can make a Thousand people do this how do I make a Million people be able to access this uh Could you talk a little bit about taking An idea and bringing it to you know Going from the the lab to the the living Room kind of

Maybe I know you work on this like Specifically back again too too and we Actually have a great recent example um We released motion control recently and Uh director's mode so Um working very closely with the Community and the feedback and creators Where with Gen 2 right now you could do Text to video image to video so you type A prompt like beautiful waterfall you Get a video back and some of the Feedback has been around we want more Control about what's generated how uh How the different scenes are coming Which way we can put the camera in so Getting that feedback then we have an Amazing world-class research team uh Getting them to ideate on it and this is Where I think because we're an applied AI like full stack company from that Moving quickly to getting it in a design Having our front-end team be ready to so That when people are navigating that It's in an intuitive way they're not Really worrying about the engineering And the research they're just thinking I Want to create and I want the camera to Go this way or I want to increase motion And then of course on the back end Making sure sure that we can you know Serve the incoming request scale out Properly run as fast as we can optimize The way we're serving models so I think That's an example of where we can go

From we have something that's released We hear feedback we can engage with our Research team and once we have that Quickly you know get it out into the World and get more feedback and go on From there good and Ty in 12 seconds or Less can you tell me how you build Absolutely million dollar company I wish somebody could tell me that in 10 Seconds or less Um yeah I I we are yes we launched the Platform two and a half months ago live To to paying subscribers we're at our Closed beta back in March and so we just Launched the cloud platform two and a Half months ago and it's been a really Exciting time I mean really fun and and Uh I think we're definitely uh new to Having a product live out in the market And uh it's all really exciting and for Both of us we're first-time Founders so I think it's weird with a startup you Know you're I think you you can Sometimes only see the light at the end Of the tunnel and you just have like Insane tunnel vision and sometimes you Forget like where are you where you Started so it's it's fun to look back at The scale I definitely did not answer Your questions I think I think it's crazy you're Thinking these things at 16 years of age I mean because you said 10 years ago Right I mean you know like I always

Think of the very different things when I was 16. All right well uh thank you very much For joining us it's been fascinating uh And I hope we get to talk again soon Yeah thank you thank you [Applause] [Music]