The Four Grand Challenges of Robots in the Home

Key Takeaways

good morning my name is brandon rohr and i'm a data scientist at irobot i'm really happy to be here because we get to talk about two of my very favorite things data and robots while robots are inherently cool they've shown themselves to be particularly useful for handling the three ds jobs that are dirty dull or dangerous most household drawers fall into the dirty and dull categories making them ideal candidates we've made a lot of progress toward reliable home robots in the last 10 years but often that progress shows just how far we have to go there's still an enormous gap between what a robot and a human butler can do we're a long way from rosie of the jetsons i'm going to cover what i see as the four largest challenges and what a path forward looks like through each of them i'd like to emphasize that these are my own opinions and don't necessarily reflect the position of irobot in any respect the four main challenges a robot has to overcome to be effective in the home are interacting with humans completing tasks determining affordances and ensuring privacy or in the form of questions what does my family want what should i do what can i do and how do i protect my family's data we'll step through these one at a time but privacy is the foundation for everything that follows without trust in our home robot assistance any other value they provide just won't be worth it data security and privacy has become a popular topic but we're still figuring out as a society how important privacy is to us and how to protect those most vulnerable to harm from the lack of it ensuring user data protection is just the first step i'll take it as a given that we and the companies we work for are acting in good faith to protect and respect our customers information but a more subtle and equally important issue is the inadvertent release of sensitive information to show what i mean here's a story about taxis in new york city in 2014 a data analyst named chris wong requested a year's worth of taxi data from the city of new york the medallion numbers of the cabs and the hack license numbers the drivers had all been encrypted on the surface there was no information that could tie a record in the data to a specific car driver or passenger however a software engineer named vijay pandurangan realized that the identifying numbers had been encrypted using md5 a weak encryption algorithm and he also figured out that if you know the format of the original data for instance that it's a six digit number then you can brute force reverse engineer the encryption another software engineer named jason hall uploaded the de-anonymized version of the data set now making what was an anonymized data set available for searching by medallion number and hack number the data now revealed the cars and the drivers involved but so far had no information that could tie it to individual passengers then a graduate student named anthony tucker was able to make a connection with photographs of celebrities taken by paparazzi celebrities in new york are often photographed entering or exiting taxi cabs and these photos often capture the taxis medallion number with this final link in the chain a collection of celebrity cab rides became publicly available data that was intended to be private and some data that was never even recorded suddenly became part of the public record through just a little cleverness this is an example of how compliance with privacy practices is not sufficient on its own for protecting data it takes careful thought and deliberate action to keep the data safe with tools like aggregation the addition of noise and avoiding unnecessary data collection altogether there's another more prominent example of unintentional information released more recently in 2017 the fitness tracker company strava released a world map showing an aggregation of all its users activity it showed every recorded user location ever uploaded the resulting maps were beautiful visualizations and there was nothing in them to tie a location to an individual user or even a particular time however an analyst named nathan reusser noted that u.s military bases in the middle east some of which are covert lit up clearly on strava's map they showed the outline of roads and jogging tracks and their brightness gave hints at how many people lived and exercised there to their credit in the subsequent version of the heat map strava plugged this information leak by omitting low density locations where activities of individuals or a small group might be represented requiring a sign up to be able to view street level data and allowing individual users to create geographic privacy zones or to opt out of the heat map data collection entirely i'm proud to say that respecting and protecting customers privacy is a high priority at irobot both in policy and in practice our data handling principles focus on minimization doing only what directly benefits our customers we only collect as much data as necessary collected data is de-identified associating it with individuals only as necessary the data that we do collect is only processed to the extent necessary and is only retained as long as necessary and the only employees that have access to it are the ones that need it when i say as much as necessary i mean necessary for the customer's benefit if it doesn't give the customer a better product or a better experience then it's not worthy of being collected processed or stored the mindset of minimizing our data footprint is unusual in industry and it makes an environment that i'm proud to work in it's been said the data is the new oil i prefer to think of customer data as the new uranium it's undeniably powerful but if not treated carefully and protected it can end up causing far more harm than good to everyone involved data minimization is a principle that robots of all sorts can use to protect their homes as robots get woven into the fabric of our lives at home it's not enough for them to be compliant with privacy laws and policies although that's an excellent start but home robots will sit somewhere in the middle ground between appliances and pets between tools and assistance they'll see us at our worst at our lowest and most vulnerable for them to be effective we will have to trust them part of this comes back to limited data collection and limited processing for instance in polite society there are observations that we refrain from making imagine a smart lighting system that can detect activity room by room so that it can turn off unused lights if it focused on patterns in the bathroom light and based on recent anomalies offered to order you a fiber supplement not everyone would appreciate that that's an instance of excessive observation and processing that destroys trust rather than builds it it introduces a creep factor this is top of mind for us at irobot we're supported by our customers and our survival depends on trust in order to build that trust even beyond what's required by a sensible and ethical privacy policy we give the user transparency in control customers can request their data and they can correct any inaccuracies or misrepresentations customers are notified and have to explicitly opt in to any data sharing or transfer outside the company these approvals for opt-ins stand alone and are not buried in or tied to end user license agreements customer trust is a very high bar and one that we have to meet every day to illustrate how important trust will be between us and our home robots consider a medication dispensing robot note that this and all the other hypothetical robots presented here are to show how much work remains in these grand challenge areas any relation to actual robots real or planned is purely coincidental this imaginary robot has hoppers for liquid medication and pills it can sense and reorder medications as they get low it can even offer reminders to individuals who are in danger of missing a dose imagine the convenience i have a shih tzu who is currently on nine different medications a device like this would be very convenient now put your black hat on if you had access to all of the information this robot collects how could it be misused medication and its scheduling would allow you to make a very good guess at users health issues this extremely sensitive information could be horribly misused by unscrupulous insurance companies or potential employers it could also be used to target health care related advertising to the most vulnerable populations this robot could only be useful if you were able to place a great deal of trust in it now let's consider a dishwashing robot this seems more benign and who wouldn't love an automated solution to the problem of dirty dishes however a clever dishwashing robot has access to plenty of sensitive information too like what did this household eat and when how many people are there do they cook or eat pre-packaged food or carry-out what products are they likely to buy where do they shop what might their health issues be how much do they spend on food it's easy to see that with an aggressive data collection and processing environment the dishwashing robot can be made to reveal much more about you than the fact that you don't like washing dishes this underlines the importance of being able to trust any intelligent device you bring into your home for a robust home robotics ecosystem this will be an absolute requirement affordances is a fancy robotics term for what can i do if you think about text-based computer game sometimes figuring out what all your options are is one of the toughest pieces of the puzzle if the game says you're in a room with a table and a chair and a chest what are your options sit on the chair try to open the chest if you find it locked smash it on the floor break the chair leg and use the table using the table and then use the splinter chair leg to pry open the chest determining your affordances can be the biggest part of the challenge the same is true of home robots for instance doors look a lot like walls how do you know whether a wall is a door whether it pushes pulls folds or slides how do you know which walls are cabinets whether they slide out or pivot to the side it doesn't help that built-in furniture can sometimes be made to look exactly like a wall how do you know which objects can be moved or which can be navigated around determining affordances is a fascinating robotics problem it's an example of something that tends to come naturally to humans so naturally in fact that we have a hard time posing the problem for a machine to solve a great example of tackling a tough affordance problem is robotic towel folding peter abeel a university of california berkeley professor and his team taught a two-armed robot called pr2 to fold towels towels are tricky affordance problems because it's not obvious where to grab and what to pull in order to get the results you want towels are supple and change shape whenever you touch them it's hard to make an explicit set of instructions for how to fold a towel because it's never in the same shape twice professor abile and his team tackled this problem using a series of images from different angles to make a 3d model of the towel based on that they were able to identify corners and using the corner locations they were able to exercise a standard routine for folding towels it was a very impressive accomplishment their achievement was limited though it assumed a rectangular towel but most laundry consists of things more complex than that also the robot was very slow it took 20 minutes to fold a towel in its first incarnation it eventually got down to 90 seconds but still much slower than a human folder an even higher stakes case for determining affordances is identifying pedestrians self-driving cars continually have to answer the question can i drive this way and the consequences of getting it wrong are very high any location containing a pedestrian is an automatic no-go zone determining pedestrians locations is a critical part of self-driving cars affordances for the most part this is a problem that's been solved well deep neural networks that specialize in finding patterns and images identify all the pedestrians in a given scene quickly and with high accuracy technical capabilities like this will enable home robots to navigate throughout the home without continually tripping over chairs and bumping into people however it should be noted that even with the massive effort that's been devoted to this particular problem failures still occur in cases where robots encounter entirely unfamiliar conditions they may fail to identify pedestrians in other cases subtle assumptions can be built into the identification algorithm for instance the pedestrians will only be in crosswalks not jaywalking this may be true in some communities but it's certainly not true in boston and any algorithm that relies on assumptions like these will certainly fail when building robots that need to work well in homes all over the world we can't afford to make convenient assumptions about what they'll see to a large extent they'll have to figure it out as they go working with floor cleaning robots has given us a lot of experience determining affordances one strategy that has proven useful in physical environments is having a variety of sensors irobots roomba and brava have cameras for doing visual navigation bumpers for navigating by touch and downward facing proximity sensors for detecting sudden drops a combination of these sensors and a great deal of experimentation and testing has helped us find strategies for the robots to determine their affordances to get all of the places they need to get but not go the places they should not like down a flight of stairs one of the big early design decisions behind roomba was whether to use laser rangefinders or cameras for navigation irobot shows cameras in part because they're better at determining affordances lasers are good at finding how far away the nearest thing is in any direction but a laser can't tell a bed skirt from a wall it doesn't know what can be pushed out of the way and what's a true obstacle determining affordances will continue to be a challenge as home robots extend to more interactive tasks imagine a robot whose job is to tidy up to put things back in their place shoes toys clothes pillows chairs papers it will need to know how it can safely lift an object which objects can be scooted along the floor what might spill the size and shape and compliance of objects will vary more than can possibly be represented in a set of explicit rules it will fall to the robot to figure out what in its environment can be manipulated how to move things from one place to another and how to stack organize pack and nest these items consider the still higher bar of a pet grooming robot here the environment in question is in motion neither rigid nor entirely fluid is sensitive to pressure and sudden changes in pressure and may actively be trying to escape brushing through fur requires applying consistent pressure in the right direction while responding gently but firmly to snags interaction with the moving compliant and possibly adversarial environment is definitely a distant goal for determining affordances now even after you determine all of your affordances challenges remain at the next level up knowing what job needs to be done and knowing when it's complete is a tough problem in its own one solution to the challenge of task completion is to define your problem very carefully machinery for harvesting grain runs almost entirely on autopilot its job is straightforward cover every part of the field at least once this clear task definition allows for efficient path planning and straightforward execution the system knows that it's done as soon as it has visited every point on its planned path this approach works well when there are no unexpected glitches no inaccuracies in your map but if a flood left standing water or if a fence had been relocated since the map was last updated these would result in a discrepancy between the planned path and the real world probably something unpleasant would happen in addition to narrowly defining the task robots can achieve high rates of achievement high rates of task completion by working in a carefully structured environment this is a robotic retrieval system for books in a library at the university of utah seekers log their requests and the robot pulls the bin with their book in it and brings it to them books can be stored in any location and in any order allowing for efficient use of space and storing frequently accessed books closer to the access point the entire system is closed to humans when in operation and carefully structured each bin is the same size and shape the robot handles bins which are rugged and uniform rather than books which are varied and can be delicate all of this together means that uninter unintended interruptions to each retrieval mission are rare this is also the approach used by warehouse robots floors are kept smooth flat clean and free of obstacles bins and pallets all come in a standard size and are placed in specific locations the warehouse is closed to the elements lighting and temperature carefully maintained when it's feasible to do so creating a structured environment is a great help for task completion the only drawback to this approach is that it's brittle it's sensitive to the assumptions that everything is well controlled to illustrate a couple of years ago in one of amazon's giant warehouses a package of microwave popcorn was dropped it was crushed the liquid butter leaked out and made a greasy puddle on the floor because it was anomalous a robot came to inspect the puddle it drove through the puddle spreading the butter and causing the robot to lose traction this disrupted the robot's odometry as it counted wheel rotations to help estimate its position spinning wheels introduced errors into the process and the robot became disoriented and stuck another robot came to investigate and also slipped and got stuck this was repeated several times before the problem was discovered and cleaned up because of popcorn the downside to relying on a carefully structured environment is that as soon as that structure is violated the system can break now contrast a book retrieval system or a warehouse with a home not just a single home but all the homes in every city in every country imagine all the things that could possibly be on the floor of a home whatever you're picturing a roomba has run into it this is the challenge of task completion with home robots it is a somewhat structured environment floors are mostly flat but this still emits a dizzying spectrum of deviations irobot's contributions to task completion touch both on clarifying the task definition and on being robust to disturbances newer roombas come with upward facing cameras they find features overhead like corners of the ceiling and use these to navigate the way a sailor might use stars in the sky they also use odometry which is particularly helpful when light is low or the robot goes under some furniture and they use the front bumper to determine when they've hit the edge of the open floor with the combination of this data and some very clever algorithmic work robots can make a fairly accurate map of the space they've covered after a few exploratory and learning runs the robot builds confidence in its map and can use it the same way the grain harvester plans a mission that meets its objectives it covers the entire area efficiently and it does this while keeping an eye on its own battery charge and dust bin returning to the base to empty its dirt or recharge as necessary in order to handle obstacles or unforeseen events a few other tricks have been used once the initial plan is in place and being executed the robot continually watches for deviations between what it expects and what it observes it may lose light and lose its ability to navigate by camera it may encounter a closed door it may get a shoelace wrapped around its brush or it may high center on a threshold in order to handle everything that a capricious world can throw at it the roomba continually checks its assumptions it recalculates its position over and over again based on what it sees to make sure it's not getting off track it checks that its wheel odometer is consistent with its accelerometer readings it makes sure that the brushes are spinning as they should and if any of these things stops being true the robot initiates a fail-safe routine a sequence of corrective actions that helps get it back on track these are hard to build well based only on theory one of the benefits of having 30 million robots in homes is that they encounter some things that you would never think to test for for users that opt in and agree to let irobot use their robots experiences to improve their performance we can gather some information about what happened where and how often the patterns in these events give us clues for common failures which we can then recreate in the lab and find effective remedies for i don't know of any shortcut to this the only path i know to robust task completion is a great deal of experience there are other exciting task completion challenges to be addressed picture a home security robot a system whose job it is to monitor your home for safety and security threats for such a robot defining its mission goals is harder what should it look for how often should it check how will it know if it's done a good job a lot of tasks aren't as easy to define as discrete missions like cleaning a floor harvesting a field and retrieving a book the way we think about task definition for robots like security robots has yet to be worked out you probably have an intuitive sense for what a security robot should do and how often but you may find that when you go to reduce that to a specific set of instructions it's hard worse those rules may result in activities so structured and predictable that it would be easy for a savvy adversary to avoid my own best guess is that task definitions for robots like these will need to be more adaptive more responsive to what they experience more dependent of the quirks of the spaces they're in and how they're used it may be difficult to create a detailed specification that works well for all situations the task definition may need to become more abstract leaving more of the implementation details up to individual robots there's also a lot of work to be done in dealing with unexpected challenges during a task consider a robot whose job it is to go into the kitchen any kitchen and make a cup of coffee this task was proposed by apple co-founder steve wozniak as an alternative to the turing test the turing test was originally derived from a thought experiment proposed by alan turing and in it human judges are given the task of deciding whether the other side of a typed conversation is being carried by a human or a machine wozniak proposed the coffee test as a more interesting variant in order to carry on a conversation a robot can get by without knowing much of anything about the physical world and in fact most of the systems that have successfully passed the turing test have done so by incorporating superficial markers of human communication like sarcasm or slang rather than relying on deep understanding of the world to be successful in the coffee test a robot would have to for instance be able to identify and handle spoons of every shape size and variety it would have to handle open coffee containers possibly grind its own beans transfer liquids and figure out how to boil water even being able to locate coffee in an unmarked container is a hard problem for robots all of these uncertainties are barriers that must be overcome in order for the robot to complete its task vacuuming is certainly challenging but it's definitely not the hardest household task we can find the challenges of task definition and handling unexpected variations in a task make task completion loom large among the grand challenges of home robotics autonomous robots are fantastic but to reach their maximum potential we want them to be more than that autonomy suggests carrying out a mission in isolation without additional instruction or interaction it's a useful concept for developing theory like a closed thermodynamic system but neither of these exist in practice a home robot will be surrounded by a constantly changing collection of sounds and movement people and things some of these will be obstacles and some of these will carry important information the robot should listen to in order to behave intelligently a robot will have to navigate and interpret these colin engel is the ceo of irobot and he spent some time wrestling with this concept in a linkedin post he laid out three important attributes beyond autonomy that an intelligent robot will need you'll have to be responsive collaborative and act as part of a larger system much of human's high bandwidth social coordination takes place through speech star trek ship computers and hal 9000 helped us imagine the seamless human computer interface that speech recognition makes possible the most primitive form of this is the clapper if you happen to have seen the 80s commercials that show people happily clapping their hands to turn their lamps on and off you know what i mean when i was six years old i had a toy that operated on a similar principle a handheld clicker signaled an orange truck to switch between two behaviors moving forward in a straight line and moving backward into the right this crude remote control let me drive the toy through nothing but audio i learned quickly that the clicker could be simulated with a sharp hand clap the trick my dad used to hijack the truck more than once human speech recognition has come a very long way for example i dictated the entire text of this presentation into my phone now speech recognition is being used in automated customer service systems and to command digital assistants of all sorts performance isn't perfect but most of the time it's surprisingly good however it's worth mentioning that when performance isn't good it doesn't fail uniformly english speech recognition for those with non-native accents varies between poor and horrendous the washington post detailed this discrepancy in a 2018 report the examples of failures would be humorous in isolation but unfortunately the pattern they present is a disturbing one since it fails to serve populations that are already at a disadvantage one particularly tough aspect of human interaction is getting machines to work well with all people not just a subset but the next step after understanding what a human has said is to take action based on that to follow the instructions what a robot can do is of course limited by its construction a vacuum cleaning robot will not be able to wash the windows a voice assistant won't be able to vacuum the floor however digital home assistants are perfectly capable of doing anything that can be done with the click of a mouse they can cue up movies turn on some music call a friend even make purchases on the internet now it's possible when you're up to your elbows in bread dough to call your mom and ask her advice on gluten texture and relationships all through voice commands and when you notice you're out of trash bags you can ask alexa to have amazon send you some more this was taken to a bit of an extreme a couple of years ago a six-year-old girl in dallas asked her echo can you play dollhouse with me and get me a dollhouse alexa complied ordering a dollhouse mansion and four pounds of sugar cookies it made for a heartwarming story which was picked up by the local news in san diego the end of the story the anchor said i loved the little girl saying alexa ordered me a dollhouse and then this utterance broadcast through viewers tvs which in turn triggered a whole new batch of devices to order doll houses for their families too following instructions isn't always straightforward sometimes it matters who's issuing them it's important to have home robots do what we expect and what we see as reasonable when we ask alexa to open the pod bay doors we sure as hell want those doors to open roomba and brava irobot's mopping robot are also becoming more responsive to what their families want way this is happening is with keep out zones if you have some place that you don't want the robot to go you can pull up a map and outline that region maybe you have something delicate or a place where the robot tends to get stuck a lot you can just head off trouble before it happens and make a virtual no-go zone having this capability the robot can also start to help with keep out zone recommendations the roomba or brava can look back over their history and notice whether there are certain locations where they've had trouble in the past maybe a tall threshold or a pile of computer cables where it tends to get stuck then it can offer to create a keep out zone around this area the robot starts to anticipate what the human might want another way irobots trying to facilitate human robot interaction is by integrating with home assistants like alexa and google home now you can say alexa ask roomba to clean my kitchen and roomba will start up cleaning the kitchen floor for a while now families have been able to schedule roomba missions in advance this is an example of autonomy of telling the robot what it needs to do up front and then letting it continue to do that thing indefinitely the ability to casually initiate a cleaning job by voice is a step forward past autonomy toward responsiveness and collaboration it helps the room but take a step from being an appliance toward being a partner it's a small but clear step along the path towards seamless coordination with humans you can imagine the possibilities when robots in our home get really good at understanding what we want based not only on verbal commands but also body language and situational observations picture a robot waiter to do its job well this waiter would not only have to place and remove dishes and utensils and drinks but it would also have to watch for physical cues you would have to handle dishes so as not to touch or alarm diners it might have to interpret subtle or ambiguous verbal commands for instance one guest might say that soup was good to hint that she's ready for the next course another might say i'm good when offered a second helping the robot would also have to infer based on drink levels and food amounts and how actively diners are taking bites and sips whether it might be time to clear some plates and glasses these capabilities are well beyond the current state of the art but there are no obvious roadblocks blocks to creating them i expect that working in close physical quarters with humans and communicating with them both through speech and physical cues will be a productive line of development and will make home robots that much more helpful now imagine taking this interaction to the next level a robot that anticipates human needs before they're expressed this step takes us from moment-to-moment interaction into long-term planning and considerably more complex predictions in order to be good at preparing meals the robot will have to do more than be good at the coffee test yes it'll have to navigate a kitchen and combine ingredients to make a palatable result but it will also have to make guesses as to what humans might want to eat when and how much initially this is something that can be commanded explicitly but eventually you can imagine a robot that takes into account past requests as well as what it knows about the schedules for the household and perhaps other variables like the weather and prepares appropriate dishes based on those guesses we'll never have enough information to know for sure that its guesses will be correct and it will always be seeking additional feedback so that it can make better guesses in the future but on any given day it'll simply do its best make and execute a meal plan order the groceries it needs and prepare and serve them at the right time the ability to adapt to the latest information incorporate new feedback and still be able to make reasonable guesses along the way will be an important capability as robots gain more responsibility in our homes adjusting a home's temperature and humidity for example is a constant dance with the changing needs and schedules of its inhabitants and is further constrained by the desire not to waste energy and money like meal preparation good environmental control will be both responsive to human feedback and will also anticipate human needs these four areas ensuring privacy determining affordances completing tasks and human interaction have all shown amazing progress in the last 10 years technologies that are commonplace now were science fiction at the start of the century but there is a lot of work left to do these challenges are grand both in scope and in the reward we can expect if we solve them well these are the problems i feel energized by i'm excited to tackle them together thank you