Social World Models

Great. Hello everyone. Thank you Sheree for organizing this workshop. Thank you to Simons for bringing together two topics that have really been at the heart of my research. Building AI systems that are able capable of social interactions and social understanding of people and building models that are able to understand the world and create models of the world. So today I'll be talking about social world models. And I'd like to contrast social world models with physical world models which was mostly a discussion that we've been having this week. Right? Think about physical world models as models that um understand and recreate the physical world. They predict how actions influence physical states and outcomes in the physical world. And social world models on the other hand attempts to build internal models of people and social environments. They predict how actions that people take influence social states influence behaviors of other agents, other people in the world and interactions, how different people interact with others. So why is it difficult? Right? If you think about how we can extend this again, we have this definition where world models can be seen as states, actions leading to next states predicting outcomes. What would this state action transition look like for social world models? Well, the states would be social in nature. This can include observed behaviors of people but more importantly also internal social states. Our mental states, what we are feeling, what we are thinking, what we believe about the world, how we believe this social interaction is going. So a lot of complications arise when you start learning how to model these internal social states beyond what is just purely observed from data. Uh actions can be thought of as how people interact with others. And this interaction is often very multimodal, right? We speak to other people. We show gestures. We give food and beverages that have different odors. And this whole multiensory experience influences how people interact with others. So how do you model these very social actions? And finally outcomes prime outcomes are modeling how these interactions can affect social states, how they can affect social behaviors and outcomes of social interactions. And some of the technical challenges I believe the main challenges lies in firstly performing multiensory perception. How do you perceive the world and perceive other people through all of the gestures and all of the modalities they use to communicate? How do you interact with people using these modalities? How do you model not just an interaction with a single person but also with multiple people in some environment? And how do you do this over long horizons across multiple steps? And finally, what is uniquely social is this idea of ambiguity and pluralism. Right? There isn't just one interpretation of a social outcome or a social behavior. Each person interprets it differently and this differences interpretation leads to difficulties in modeling everything using one monolithic model. In fact, I want to give a shout out to Lena who's in the audience. Uh we actually wrote a position paper in 2024 that is still highly relevant today. It's called advancing social intelligence and AI agents uh technical challenges and open questions. We started looking at this problem of what is uniquely difficult about building AI that has social intelligence beyond physical intelligence. And we came up with these four main challenges. Ambiguity and constructs. The fact that a lot of these internal social constructs are ambiguous. For example, I might feel we have rapport when interacting with somebody else. They might think I'm crazy. The problem of nuance signals. How do you really detect very subtle facial expressions, body language, changes in gesture? How do you model multiple perspectives? Right? That's a problem of theory of mind. And finally, social interactions usually go on over very long durations. How do you build models that can continuously adapt to these social interactions? So, talk to Lena in the audience to learn more about uh what we think are the main challenges in building socially intelligent AI. So in today's talk, I'm going to cover three of our major advances in building towards AI that is more capable of understanding the environment, understanding people and interacting with people, eventually building towards this social world model. Right? The first key idea um that we've been working towards is the idea of modeling touch. And this has been recapped and mentioned quite a few times in this workshop. uh how do you model how people interact with different physical objects in the world through high resolution both spatial and temporal tactile sensing and how people interact with other people through tactile sensing. So I'll cover some of our work in creating large scale data sets and high quality sensors that capture the sense of touch. Uh secondly I'm going to talk about something that is very interesting in my opinion. It's about modeling old faction right smell is so critical to how we understand the world. Smell is so critical for how we share experiences and foods and beverages with other people. And yet AI for smell is completely underststudied. So I'm going to cover some of our work on building AI models that can perceive smells like how people can and how AI can be used to transmit smells across digital mediums to enhance social connections and improve the human experience when interacting with other people. And finally, I've mentioned this key idea that uh when you look at social world models, modeling this internal unobserved social states is really important. How a person is feeling, what they are thinking, what they might do in the future. All of these can be uh learned from observed behaviors, but it's very difficult to learn these internal unobserved states. So I'll cover some of recent work on creating uh foundation models that can generally understand people and their internal states and look at how this can drive uh world models that can better model human behavior and interaction. So firstly on the sense of touch. So we've been working on these tactile sensing gloves. If you think about a sense of touch, you want to go with a human hand, right? You don't want to go with grippers. You don't want to go with other animals. The human hand is so flexible in its way of manipulating and interacting with objects and other people in the world. So we've been creating open source platforms uh that can create tactile create gloves that people can wear on their hands. It fits exactly. So it uh given this photo of your hand against a piece of A4 paper. It automatically decizes uh looks at the sizes and shapes of your hands and it places these printed circuit boards. These are flexible printed circuit boards that contour to the movements of your hand. So all of these layers of the glove can be automatically manufactured from layers of fabric that are being cut by a knitting machine to layers of rubber that are being printed by a laser cutter to layers of these circuits that are also being cut by this laser machine. So it takes a grad student about 15 minutes to assemble these gloves layer by layer. Right? There's two layers of this PSO resistive material such that when they come closer together because of pressure or force the resistance increases and that's how the sensor reading is obtained. All of these other layers such as a layer of rubber is providing friction. This is show attaching the readout circuit to the wrist and the outermost layer is a layer of thin fabric so you can still get uh uh the feeling of feedback when you're interacting with real world objects. So this just shows the person wearing this glove and going about their daily activities like in the classroom, working and using a mouse, opening and closing a book and others. >> These are mostly coarse motor skills. >> Yes. Uh but we'll show fine grade motor skills as well. So using these gloves you can go out and perform large scale data collection where people are wearing glasses. This gives you egocentric vision of seeing what the people are saying. They have spatial audio so you can hear what they're hearing from left and right. And these gloves capture what they are feeling. And this is a calibration stage where they are going through and pressing each of the joints, fingers, fingertips, palm of the hands. And you can see the corresponding alignment between vision and touch. Uh so after you perform this calibration stage, uh making sure that it fits nicely to the user, there's no slippage, you can then go out and perform large scale data collection. So this will have egocentric vision. It will have the tactile information from both hands. And here we've [clears throat] also added a 3D hand post tracker on these gloves. This can be done using IMU sensors telling you exactly where each joint on the finger is. This can be seen as the supervision for retargeting this action onto robotic hands. So that's called 3D handles. So this open touch data set. is the largest um to our knowledge multimodal data set that contains visual information, full hand coverage of tactile sensing and 3D hampos. And you see this is a example of waving a bat in in the shopping mall. There's examples of using power tools. There's more fine grain examples of in the in the house doing cooking, cleaning, in the lab. Uh one hands and two hands. So here's just more examples. Um covers a bunch of environments, covers different actions. Actions are high level actions just picking and placing. We've also tonomize different types of grasping and different types of objects. So grasping can be seen as using your finger, two fingers, three fingers, rolling, squeezing. These are the really fine grain motor actions that uh categorize how our human hands are used to interact [music] with the world. Uh for people who are interested, we've went through a series of gloves uh various versions. The first version cost about $1,100. $100 for the tactile sensing component and $1,000 for the hand tracking. So, the hand tracking is a bit more expensive. Uh, this is open source and customizable. And there's several versions that go higher resolution in tactile sensing. So, this will cost about um two $3,000 tactile sensing gloves. These are thousand tactile sensing gloves. They are more robust and give you higher resolution, but if you want to go with something low cost, open source, and customizable, uh, you can use what we have. So now we are slowly expanding to even more fine grain tasks. Uh in this case looking at two hand tasks uh in the chemistry lab pressing one of these pipets holding a stabilizer where it should go in and you see the 3D hand post tracking key points um tactile sensing and here's just the aggregate information of which hand is showing more pressure. So we call this manipulator and stabilizer tasks. We have insertion alignment tasks. I don't know how many of you when you're trying to insert a USB into the outlet, go one way, you feel like doesn't go in, turn it around and insert the other way. Good example of tactile sensing. And finally, slip and recovery. I like this example because usually if you think about most of these models that we have today, you start with an instruction and you plan out the sequence of actions. But in many times when you start actually executing the action, you fail, right? Things slip and the environment changes. It's more slippery than you expected. So, how can you have a system where you can drop something and the other hand catch it? Right? That's another good example where tactile sensing the reaction is important and you're not just planning through vision and language instructions. We're recently extending this to really look at the fingertips, right? The fingertips are among one of the most sensitive parts of a hand. So, we're prototyping some of these fingertip sensors. Um, I should have mentioned the tactile sensor as a whole was 16 by6 resolution on the whole hand. Now we're working on these fingertip sensors that give you 32 by 32 resolution just on one fingertip, right? That brings you closer to how humans are perceiving the world. Uh, these fingertip sensors are good because they maintain dexterity. They're more lightweight. Doesn't cover the whole hand. At the same time, you can think of this exact same fingertip as being put on a robot hand and also being put on human hands. Right? So, here's just some examples of human data collection using these fingertips. And these fingertip sensors can be 3D printed. They are they have a curvature that is similar to our human hands have. There's a tightener so you can fix it to different sizes and shapes. They have passive haptics so you can still feel through this 3D printed material. And we explore a mix of both flexible tactile sensors and vibration sensors. Vibration sensors can actually be detected by very minute changes in audio as you're moving your fingertips on a surface. Here's just a quick video of these uh fingertip sensors in action. Uh here you see that they're rolling this little little piece of equipment uh that's used to screw in bolts within these two fingertips. And you can see the tactile information changing both spatial and temporal resolution. And here is the pressure sensor. So what does this give us? This gives us really fine grain information at really high spatial and temporal frequencies. And so we've been developing models that can combine these different sources of data for people to understand the world, for models to understand interaction with other people. And the core technical challenge here is that this data modalities come in at very different frequencies. You have vision that is coming in at 30 hertz. You have your classic vision encoders. You have language instructions that are coming in at one two words per second. That's the speaking rate that people usually speak at. And usually these language models that take in vision and language and output actions, they usually output actions at seven hertz because first of all the inputs are slow bottleneck by the slow speed of vision and language and these models that decode actions based on diffusion models are also slow. So there's main bottlenecks to some of these vision language action models and tactile what tactile gives us is really high resolution information right being sensed at very high frequencies up to 100 even 200 hertz and you don't want to bottleneck tactile by going through these slow vision language encoders and action decoders. We've been developing this approach that essentially allows this tactile at high frequency to model the residuals. So tactile can go through a separate encoder based on CNN's and a separate action decoder which is much more lightweight based on GRUs or recurrent models. And you can think about these as filling in the gaps that vision language models and action decoders currently don't give you. So this then brings up these temporal gaps and allows you to decode actions at a much higher rate right up to 700 hertz instead of seven hertz. And here are just some examples. Uh so we started with this vision language action model called VR 3B. We fine-tune on gigahands which is a vision only. So people are wearing cameras and digitizing how they're using their hands. This gives you pretty bad mean squareed error uh because there's a distribution shift to this open touch data set that we created and it's slow. You can then start fine-tuning the open touch to minimize the distribution gap to this data set. So now you see better hand key point tracking but it's still slow because it's still relying only on vision and language. It's about seven hertz. And once you start using vision and language plus this tactile editing you get uh even better m squed error with higher frequency about 700 htz. So that's a sense of touch, right? Building sensors, creating large scale open source data sets. Open touch is publicly available and training models that can seamlessly combine vision, language, and tactile information at various sensing frequencies. So we're also super excited about the sense of smell, right? The sense of smell is almost completely underexplored in modern AI, but yet smell is so important. It's how we look for delicious foods and beverages. It's how we avoid dangers. It's how we share connections and memories with other people. So, you've been working on general purpose AI models that can both sense and transmit smell. I'm going to show a quick demo over here where you have three spices and you've portioned out a little bit of each spice and you bring them towards this smell sensing platform with an AI model running on the edge that is tracking the gases that are released by these spices. These sensors are VOCC sensors. These mean volatile organic compounds essentially volatile gases based on carbon, oxygen and hydrogen that are released by different foods and beverages. Uh so this model senses these different VOCC's and it's measuring the similarity with what it has seen and recorded in this prior database. So it spends the first 60 seconds becoming more confident in the similarity with the spice oregano and it spends the last 60 seconds being confident in that prediction whereas the similarity with everything else. How many classes of sense of smells do you have? >> Right now, this demo is showing four three spices plus ambient. And I'll show you the the full class later. >> So, here is just um the video in static form. The substance, the sensor in this case is a commercial sensor cost about $10. It senses six VOCC's uh like carbon monoxide, ethanol, alcohol, nitrogen dioxide, mixtures of CO, H, and N. And these are temperature, humidity, and pressure sensors which are also known to change in the presence of a substance. And all of this is just time series data. So it's how time series data is changing based on the concentration of the gas that is detected. So to us, you can then start using your favorite multimodal and time series analysis models to learn representations and make classifications on what the substance is. Of course, data is key. So we curated a large data set that we call smell net hoping to catalyze the same effect that imageet had on computer vision to encourage people to work on AI for smell where we collected 50 substances 10 types of nuts, spices, herbs, fruits and vegetables. We put each of this in a small containment container bring it to the sensor and let it sense for about 10 minutes. That gives you 600 data points at one hertz. uh this logs your various VOCC's and other environmental conditions that change within that nearby environment and we re uh we we repeat this over multiple environments. So sometimes indoors, outdoors, sometimes when people are walking around it, sometimes when the AC is on or when the fan is blowing so that we can get robust readings over many different conditions. We then align it by annotating a textual description of what it is. GCMS which is a chemical representation of the substance and also a photo. Right? So one method that we found to work really well is that these VOCC gas sensors that we work with are pretty low resolution. They only sense about 10 volatile organic compounds. There's 10 sensing regions. They are not very selective and sensitive, but they're portable. They're cheap and they run in real time. Some of these other more fancy equipment and people in chemistry study this. This is called gas chromatography or liquid chromatography. Uh it's the stuff that you did in high school where you broke a substance down, you added some liquid, you put a piece of paper and you saw the the colors go up the piece of paper. Um so this called gas and liquid chromatography. This is much higher resolution. It detects all combination of molecules in the substance. Very sensitive. But these machines are not portable, very expensive and take hours. So what can you do? One approach to get really good performance is to do crossodal learning. Right? This is a common problem that you see when you have real world data modalities with different trade-offs. You have the low resolution sensor. You have this high resolution information that is paired during training and you try to learn a representation that is using the high resolution data to supplement the low resolution data. You can do this by either aligning data modalities using contrastive learning. So you learn an embedding space where your gas sensor representations are nearby your molecular um representations or you can also do this via translation by using the gas sensor to predict the higher resolution data and we found that helps a lot right there's uh using contrastive learning to learn these align representations allows you to get the best of both worlds and we also found that you can actually create approaches that use temporal differences Sometimes these absolute sensor readings are very sensitive depending on a day. So learning these differences in readings, the relative changes in readings and using that as input to your sequence models work better. So once we've trained this general purpose model, we started fine-tuning it. Uh here's one very exciting application. Can you use it to detect allergens? Right? People who are allergic to peanuts often worry whether their food was cooked in peanut oil, whether there was contamination and sometimes you cannot see it visually. So we've trained a specialized peanut detector that does not work all the time. So not FDA approved, but you can bring it to some candy bars, peanut butter, and it can start detecting the presence of peanuts. Uh to wrap up this discussion on smell. So we talked about sensing. What about transmission? Right? smell transmission, being able to send smells to your friends, that would really enhance how we communicate with other people. It would enhance advertising and it would enhance how we experience the world beyond just seeing things on the screen. So, we uh as you as you so as you show, we're working with some of these spices and you also have some of these essential oils. These are essential oils that you can buy from any supermarket. They have the flavor and essence of various fruits and vegetables. And you can use some of these sensors to detect how similar these essential oils are with the actual substance, right? And for the essential oils that are similar, we've been prototyping some of these wearable devices that can contain six containers on the left and six containers on the right. Each of them you can put an essential oil inside of them. And this allows you to recreate by releasing these essential oils an approximation of the food and beverage that you started with. M how is this done? Uh in fact we find that a lot of multimodal language models has a good amount of information about how smells mix with each other. So imagine you start with a a feed a salad, a photo and a description of the salad. So you any vision language inputs you give this to a multimodal LLM a vision language model and you prompt this vision language model to generate how this smell might decompose into a mixture of 12 base smells. Right? 12 is chosen just because of the current hardware that we're using. It fits six smells on the left and six smells on the right. Now, you got to choose these six 12 smells carefully to cover this whole spectrum of smells and smells that are likely to interact with each other. Right? So, by prompting the model to decompose this input into your 12 smells, the model is able to say, "Okay, this should be eucalyptus and some amount of clover, some sage, some strawberry, some onion." Right? Once you get these mixtures, it is then programmed and sent to this device which releases these substances in that order. Yes. >> How do you figure out this basis of this little space like yours? >> Yeah, great question. So, we went through a long process where first we look at literature, right? What have people found about different smells and different bases of smells? Sweet, savory, sour, burnt, smoked, and fresh. That was kind of a taxonomy that um people used to describe the language of smells. And then we got expert perfumers to come to our lab to look at the essential oils that we had because these expert perfumers they literally mix smells for a living. So we asked them which ones were under represented, which one were over represented and eventually we came up with this list of of 12 smells. And you can play with it. Some of this smells like cheese, meat, umami, spicy, sour, uh fresh >> and and you still generate those 50 uh classes of smells. You don't have uh fine grained continuous or do you smell generation? >> Yeah. So in smell generation here we have mixtures of these 12. So it's different from the 50. The 50 were food substances like you know orange, apple. >> Um they do not exactly overlap with these 12 base notes. We think of these as RGB of smells. And of course, a lot more work to be done to figure out what the RGB of smells are. And the other 50 are just basically object categories or food categories that we're working with. >> So, what variety of smells can you generate? 12 substances. >> Yeah. So, um we've been trying with a with a bunch of smells. Um we try with basically your classic foods. Cheeseburger, salad, pizza, [clears throat] chai, coffee, tea. they um doesn't work all the time. Uh and I'll show you some of these results. Um so we basically compared over here user studies across multiple uh many participants about 30 participants and we compared a human mixing the smells. So the human access to those 12, how will they mix it to reconstruct the smell? A AI approach that does zero shot prediction of how the mixture should be and also an approach where a human gives a further feedback. Right? Um here I was showing that sometimes people are overly sensitive or under sensitive to certain smells. They can give feedback uh to the system and then starts learning from that feedback and tailoring it to the user. Uh so you find that in some cases it was worse than a human expert mixing a smell. In many cases it was better with AI and even better as you give it a few steps of refinement. And just for some qualitative studies people showed um in this case they really appreciated the accuracy of the AI mixture that they wouldn't have thought about it um representing abstract memories. Someone used this and said, "This smells [clears throat] exactly like the beach we went on for a honeymoon. Can we have this exact combination of smells to bring back home?" Of course, the model doesn't know where they went for the honeymoon, but it generated a smell that was able to trigger the memory uh that they actually remember their honeymoon. >> Yes. >> Smell the concentration is also a big factor, right? In the the way when people feel the uh classify the the smell. Is is this a problem in the uh that the concentration can the same ingredient but different concentrations? >> Yeah, to be clear >> completely different different smell, >> right? So here the model takes in a vision and language inputs or any prompt that a user gives it decomposes it into a set of 12 numbers that measure the concentrations of each of the 12 smells to release. And in this case concentrations are durations. So it releases something for 10 seconds versus 1 second versus 5 seconds. So it's able to to mix and match um according to different durations this mixture of 12 base smells. >> Yes. >> Just understand. So so the zero shot is basically because people have described smells in language so much that the system is kind of able to figure out what kind of of these smells should be in the smell you're asking for. Is that kind of the so basically through language you could figure out smell in some way I thought it's yeah but but there also context dependencies right smell but you have like the same molecule can smell like really pleasant really bad depending on the context I forgot which one it was but there are examples like >> yeah yeah so so uh exactly so we believe that this zero shot works because we have you know lots of recipes cookbooks that describe smells so there is some latent information within natural language uh and of course um you need personalization Right? That's why we added this further refinement step upon which as people interact with it and refine it by saying this is too sensitive for me or this is these two don't mix together. This gives you the real world interaction data which you can then use to further fine-tune the model which um as we're showing here going from without learning to the one that it learns from the refinement from the users interacting with the system uh gets you better performance. >> How much time do I have? >> Couple minutes. >> Okay, let me just wrap up. So, um, so I'm really excited about this idea of smell, sensing, and transmission. So, you think about it, what other form factors could there be? Uh, right now we have these necklaces that people wear. They shoot something up to your nose. It can be inside the nose. It could be on the phone. So, think about actually building AI systems that can go through the images that you're scrolling as you're on Tik Tok, on Instagram, generating smells in real time. How will that change the human experience? And also um after we put this out very exciting a couple of students were trying to put this in the car right the car is a very nice enclosed environment where you want to regulate the emotions of people right get them to be alert when they're sleepy get them to calm down when they have road rage like people in Boston would do anything to make driving a better experience in the city so a couple of students actually put this and see whether uh they can be used to make driving better some it was picked up and the F1 driver actually came to this driving simulator we developed to see whether sense can make you drive better. Uh and lots of open research questions. How do you connect smell sensing and transmission? We haven't been able to do that yet. Right now this uh transmission was still done using vision and language inputs. How do you do smell removal? If you want to do this in real time, you have to remove previous smells. And that still poses both scientific engineering challenges. How do you combine smell with other senses? And of course would love to chat with neuroscientists who are experts in this space. [clears throat] So to summarize um I'm really excited by world models that go beyond a physical world to really understanding and interacting with people. And what's critical for that? It's critical to understand human sense of touch and how they interact with objects and other people. It's important to understand old faction as a new modality of understanding the world and using transmission to convey changes in emotion and behaviors. I wasn't going to get into this, but we also do a lot of work that aims to understand human internal states from their explicit behaviors. Thanks. Happy to take any questions. [applause] >> Yeah. >> Just wanted to give a comment regarding your last remark. So studying memory, emotion, and then old faction, all that. If you I mean search and uh try to connect your entropies with the field of associative conditioning. In associative conditioning these smell and different modalities are connecting in the brain and it's a great starting point for you to integrate all of these uh different modalities from a neuroscience. >> Sounds great. Thanks. >> Okay. Thank you so much. [applause] >> Yeah, I will too. Okay, next we have Chris Kim from uh UC Berkeley and his work together with Bruno. >> Great. Can everyone hear me? All right, awesome. So, yeah. So thank you Sherry and