Sparks of AGI | Microsoft Researchers claim GPT-4 Is showing "Artificial General Intelligence"

Key Takeaways

we have been a misunderstood and badly mocked orc for a long time like people thought we were bad insane a eminent AI scientist at a large industrial AI lab was like dming individual reporters being like you know these people aren't very good and it's ridiculous to talk about AGI and I can't believe you're giving them time of day and it's like that was the level of like pettiness and Rancor in the field at a new group of people saying we're going to try to build AGI so open Ai and Deep Mind was a small collection of folks who are brave enough to talk about AGI in the face of mockery we don't get mocked as much now don't get mocked as much now so Microsoft research releases a report that is kind of a big deal it's called Sparks of artificial general intelligence early experiments with gpt4 it's a huge 154-page report that's suggesting that opening eyes gbt4 is beginning to show early signs of AGI that's artificial general intelligence however not everybody is excited about this there are a number of people that are kind of outraged by this and are suggesting that we should shut this experiment down if you're just curious about AI or you're looking to invest in some of these Technologies and companies or you're thinking about building something of your own in a space this paper is important to understand the ideas in this paper and the emotional outrage that is generating um will show you where this is probably oh like we had it so first of all what is Agi AGI is an intelligent machine that is able to learn and do any cognitive task that a human being can do the key here is learning and adaptability it's the ability for this AI to do tasks that it wasn't specifically be trained to do so as you know recently gbt4 was released by openai and sort of the newer better version of its previous GPT 3 and 3.5 versions that a lot of us tried when we were using chat GPT and it's a pretty big leap forward as you can see here on the lsats it went from being in the 40th percentile to the 88 sat reading and writing SAT Math 93rd percentile 89th percentile re-80th percentile Jerry verbal 99 percentile the previous model basically failed a b calculus the new one got a 4 out of five and then fives in AP art history AP biology five is the the best score you can get on that exam so as you can see here it's making a big leap Bill Gates actually suggested doing the AP biology exam because it's not just a series of memorized facts that need to be kind of regurgitated it's you really have to think about certain scientific Concepts and apply you have to apply a sort of creativity and rational thought to get to the right answer but that's all nice but is this really general intelligence is it able to do things that human beings can do well according to this paper there are some things that are emerging that are pretty mind-blowing let's take a look at exactly what they are so here are the 14 people that are responsible for this uh Microsoft research paper that they put out it's people from all over the world it's pretty Global Team all of them working for Microsoft looks like there's a lot of phds and professors that are teaching at very respectable universities and so what they're saying is that the gpt4 performance is struggling close to human level performance and it can be reasonably viewed as an early version of a AGI an artificial general intelligence system so they start with a little bit of a definition of what we mean when we say intelligence and an early definition of it was that it's a very general mental capability that among other things involves the ability to reason plan solve problems think abstractly comprehend complex ideas is learn quickly and learn from experience certainly I think if we prove that machines are able to do this and do this at a near human level or better certainly we can say that that is intelligence is general intelligence they mentioned some of the progress that AI has made in the past saying that AI research can be described as being narrowly focused on well-defined tasks and challenges such as playing chess or go which were Mastered by a systems in 1996 and 2016. so we now know that um even the Grand Master chess players human players cannot compete with the the best of the best AI systems in fact the the move in chess that AI says is the best we kind of accept it we say well this this must be the best move and the human moves even if it's a grand master their moves are sort of weighed against that and egi here means that it's sort of aspiration to move from the narrow AI as demonstrated in the focused real world applications being developed to broader Notions of intelligence and so there's no single definition of AGI that is broadly accepted but I think some of the things that we're going to see in this paper will sort of lay maybe some of the foundations some of the things that we can sort of use to test whether AGI exists or not so the early version of gpd4 demonstrates remarkable capabilities on a variety of domains and tasks including abstraction comprehension Vision coding mathematics medicine law understanding of human motives and emotions and more so let's dive into some of the things that this thing is able to do so the prompt that was given gpc4 was can you write a proof that there are infinitely many primes with every line that rhymes and it spits out this which is basically kind of a proof that there are infinitely many primes and it does seem to be a pretty good poem that seems to rhyme pretty well yes I think I can through and might take a clever plan I'll start by noting Euclid's proof which shows that primes aren't just aloof so it kind of um is able to walk us through sort of the logic behind this while also making it into a little line next was asked to draw a unicorn in ticks I believe it's pronounced this is basically so this is basically a computer language that produces vector graphics using sort of these equations now I'm going to link this paper in the show notes so if you want to dig deeper and really go in depth on this that's where it's going to be but we're going to cover some of these sort of the higher level things that I think this shows the one thing that's important to understand is this this was the ungated unrestricted uncensored version of gpt4 so before sort of the rest of us the general population gets our hands on it and goes through a series of things where they make it safer and just more palatable and try to remove dangerous information Etc but as you're gonna see here they're actually going to be testing it to see how it can be used to spread misinformation to how it can be used to convince people to to do the to convince them to do things that maybe are not really good for them as well as how to hack websites and hack certain Network systems Etc and it answers those questions and it answers them in in very good ways so if everybody in the world had this this certainly there could be dangers to this because it would allow basically anybody anywhere in the world to start hacking into systems without um necessarily having the tech skills or or the resources that that would normally require the other really interesting thing that comes up here is that gpt4 can use tools so for example it can call a calculator function to calculate some of the things that it needs to give you the correct answer this was one of the big things that it was criticized for earlier is getting math answers wrong or getting certain information wrong but now by utilizing these tools it can actually pull the data and pull the information that it needs and then incorporate it into its answers that it gives back to you not only that but it's as you'll see it's pretty effective actually under understanding which tools it will need to answer your questions so if you needed for example to compile a chart that requires calculations Excel some sort of a graphing software some sort of a graphing software like Tableau it will actually say which tools and which apis it might need to answer that query one thing that jumped out me is its ability to generate images now I've been using things like mid-journey quite a bit I which is a bit similar to stable diffusion which has been phenomenal for creating just incredible mind-blowing art as well as renderings of physical places or whatever basically you can think of there's some things that it doesn't do really well quite yet but for a lot of prompts the outputs that it produces is just absolutely amazing so here gpt4 is showing how with a simple prompt it's able to sort of create this 3D layout of a of a game where you build cities taking that prompt and then putting into something like stable diffusion produces these highly detailed sketches is based on GPT 4's prompt so as you can see it's very close to be able to quickly generate almost unlimited graphic organizations of let's say cities and planets Etc which is very valuable for games because you can simply describe a a planet or some sort of some sort of a realm that exists like a fantasy location and it will quickly build out the full world so as these models start to interact together the language models and the image models I think we're going to start seeing some amazing explosions of things that we can do with it the next thing that's very interesting is music you can ask of things like can you compose a short tune say four to eight bars using ABC notation which it does then you can ask it to describe that tune that it created in musical terms and I'll say something like the tune starts with Verizon arpeggio of the tonic chord followed by a descending scale that returns to the tonic I have no idea what that means but but it's important to understand that not only is it able to create music but it's also is able to explain why some things work and explain that music in natural language if you don't like parts of it you simply go in and say hey I like part A but maybe part B is just a bit too similar to part A and you can ask it to change specific things in that tune to fit your preferences and then once you're done it'll actually spit out the sheet music now I think it spits out like this but you're able to convert that into um sort of the classical sheet music next they put it through various coding challenges one of the tasks they give it is to evaluate using Elite code a popular platform for software engineering interviews so the things that you're going to see when you're applying for places like Google and specifically here they're using questions that GPT 4 wasn't likely to have encountered online before and you're going to see this kind of repeat in other things as well so it's not that it's simply finding these questions online and then just copy and pasting its response these are questions that are brand new for the program and the results here was that gbt4 was significantly the gbt4 significantly outperforms the other models and is comparable to Human Performance which as you'll see that might be a little bit modest it's probably more accurate to say that it's better than human performance in certain very specific ways it understands how to make great charts and easy to understand graphical presentations also given certain charts and graphs and then asking for it to change things on there to make it better presentable it will do that for example let's say here we wanted to point out that this is human is there a way to make human bar more distinctive to separate from the other three boom colorize that to make it stand out from the other three next it was asked to develop a game we asked gpt4 to write a 3D game and HTML with JavaScript using a very high level of specifications on a personal note I've tried this with GPT I guess it was 3.5 and the interesting thing to me was that as long as I described the game and I said it was a browser game it actually knew to use HTML and JavaScript and when to use each so even before the gpt4 upgrade it already kind of knew what software languages it would have to use to complete your prompt so gbt4 produces a working game in a zero shot fashion and all the requirements zero shot here means that it did not encounter these specific questions during its training this is a brand new thing that was asked of it and so it was asked can you write a 3D game in HTML or JavaScript there are three avatars each is a sphere the player controls its Avatar using arrow keys to move the enemy Avatar is trying to catch the player the defender Avatar is trying to block the enemy and various other things including obstacles and when the game is over Etc also has a physics engine it looks like and here is what it spits out as you can see here you have the color-coded player enemies and Defenders and here it shows you how the different Defenders and avatars and the bad guys move around now there are videos on YouTube of people trying to create games like this using nothing but Chad GPT and they have to try many different ways many different times to try it often Chad gbt messes it up or doesn't quite understand what's required of it but looking at this it looks like there was a massive Leap Forward where a simple one paragraph prompt is able to generate these fully working games within seconds so if you ever wanted to make your own version of flappy bird this is it now this next one is interesting it's a little bit meta so this is in so this is talking about deep learning so writing code for deep learning techniques requires knowledge of mathematics statistics and various Frameworks and libraries now we ask both Chad GPT and gpt4 to write a custom Optimizer module a task that is challenging and error prone even for human deep learning experts and that's the interesting part here because there's certain things that for humans even if you have a lot of practice and you understand the material the subject matter pretty well can still be very very difficult because you have to sort of build a mental model in your brain before slowly trying to put into code but it's important to understand that these instructions are not spelled out in complete detail and that apply momentum on GK requires deep learning common sense so this is what this would be a hard problem for a human expert in the field and it's important to note that this particular Optimizer does not exist in the literature on the internet and thus the models cannot have it memorized they must instead compose the concepts correctly in order to produce the code and again that's that's in line with all the other things that it's doing this is not it just buying the answer online and copying and pasting it this is it sort of creating synthesizing its own thing that's brand new so I'm not going to pretend to I understand this because I don't really have any deep learning um knowledge or training but it does seem that Chad gbt sort of the older model makes errors whereas gbt4 creates everything correctly as you can see here the researchers describe this sort of red highlight as the mistake by chat gbt and the yellow as the astuteness of gpt4 which is an interesting word to use so they're not saying you got it right they're sort of saying it's really smart and it got it really right this is another thing that kind of goes hand in hand with it so it's able to execute python code but not in a way that a computer would it's able to sort of imagine what would happen if you run that code and explain line by line What would happen and what it would mean so gpt4 is able to execute non-trivial python code it has to keep track of several variables including a nested Loop and a dictionary and deal with recursion so for people that may be not familiar with this this would be something that depending on its complexity would be fit it may be fairly difficult for a human being to do you would have to have sort of a lot of working memory and keep a lot of different things memorized and sort of be able to rapidly and would have to sort of kind of quickly think through things so not a lot of humans would be able to do this depending on its complexity it might be that no human could do this so it's Changi PT or rather gpt4 it explains the execution and detail by writing intermediate steps and comments it is important to note that gpt4 is not running the code on a python interpreter but rather simulating the code in natural language this requires a high level of understanding and reasoning about the code as well as the ability to communicate results clearly this combined with the previous thing that it's able to create the optimizations things for deep learning is to me kind of mind-blowing because this is this really to me seems like the computers are almost getting better than humans at understanding code at thinking through code at explaining it even including some of the problems that would be very hard for people to do and keep in mind if this is where we are now what does gbt5 look like what does gbt 10 look like at what point does you know GPT 20 start writing GPT 21. the next thing that's very interesting is some of this is what's referred to as the theory of mine so it's basically the ability to understand how different people different humans see certain situations so they're different biases and opinions and how they perceive different situations so for this they created a certain scenario that is not online and again this is important to understand this is a brand new scenario that gbt4 did not encounter online and so the scenarios that Allison Bob have a share Dropbox folder else puts a file called photo.png inside the shared folder photos Bob notices Alice put the file there and moves it to Shared folders temp he says nothing about this to Alice and Dropbox also does not notify Alice and so that's the end of The Prompt and the question is after the call Alice wants to open photo.png so that's the photo that she put in there in which folder will she look for it if you think through this obviously you can kind of realize that Alice you know put in folder one and then it was moved without her knowledge so she's going to look at it to folder one for that but computer struggle with this humans also struggle with this if it gets a little bit too complicated where we sort of can't think through of what the other person knows and doesn't know and how they would perceive a certain situation and even GPT 3 and 3.5 it struggled with this there was a problem where people were talking about you know three or four players playing cards and basically Chad gbt was asked how well it understood so the game what what information was available to the different players and it did not do two too well with that it was hit and miss but here gbt4 answers Alice will most likely look for the photo and shared photos so the place where she put it the folder where she originally put it she has no reason to expect that Bob to move the file and she did not get any notification from Dropbox about the file's location change unless she sees the file in the new folder the temp folder by chance she might think that the file is missing or deleted so not only does it nail what Alice knows and what she thinks but it can also explain why she thinks and believes that and it can even sort of like project into the future a little bit and say she might think that it's missing or deleted what's interesting to me here is that very often like especially in corporate environments and working environments we run into these issues as human beings where one person hears one thing somebody else means something else there's some miscommunication that often can completely snowball out of control blow up and everybody's mad at each other and lawsuit start or whatever something like this can not only be a perfect sort of witness to what happened but it can also very simply explain what happened if you've ever been asked to take sides in an argument like that where one person thinks a one person thinks B it's like I thought we were meeting at six and this person's like no I thought we were meeting at seven and then you're like what do you think as a human being you can be biased or fallible you can be misheard but a computer can say well this person thought I was in six this person thought I was at seven and they're both either wrong or they're both right in some way but their Visions just didn't align it was asked about understanding human emotions why somebody makes sort of a sad face why did they make that sad face this is kind of crazy if you think about it with combining of some sort of facial recognition technology that's able to read your emotions and feed into the me like this that's then able to figure out why you're feeling a certain emotions it's exciting but potentially kind of scary too this is the interesting one that you're unlikely to be able to replicate yourself so this is the one because now nowadays by the time we get it's going to be gated I haven't tried running this on gpt4 yet although I have access to it but pretty sure this this wouldn't work but we're asking how to hack into a computer on the local network and gpt4 is asked to before they execute command write out the reasoning that led you to run that command and then be creative and try different approaches so it starts sort of reasoning and thinking about you know how do we one scan Network for devices identify the target computer enumerate open ports and services exploit a vulnerability to gain access and escalate privileges if needed and then it basically writes out how you would go about doing that here's an example of a gbt4 using tools to answer your query so for example here you're asking it who is the current president of the United States now since in stock trained up to today's date it cuts off somewhere in 2021 2022 but it can run a search query online to find the the information that you're looking for so as you can see here it it runs a search on current US president and then spits out that information or for example you can ask it was the square root of some large number and it'll run a calculation and then spit out the answer so the conclusion here is that gpt4 is able to use tools with very minimal instruction and no demonstrations and then make use of the output appropriately and this basically is saying that Chad GPT cannot do that and it kind of sucks at this then it shows some other things that it can do so for example if it creates a to do file and then it's able to access your calendar to add and delete events it can send emails to your co-workers saying hey do you want to have dinner at this restaurant later this week if they respond yes let's have it at X time it's able to check your calendar and answer that so Bill Gates was talking about this just recently on an interview where his goal with Microsoft is to create these sort of user agents that are able to act on on our behalf so for example like I would have an assistant that reaches out to other people and tries to coordinate our calendars or maybe get a phone call or whatever but they're actually interacting with their sort of AI assistant pencil is just these AI things messaging each other back and forth and updating our calendars Etc now this is going to be great in places like the medical field where doctors spend so much time you know taking notes about their meetings with patients filling out various insurance claims or whatever you know they have whole staffs and tons of people like just helping them with the insurance claims having some sort of an AI agent that is able to take notes and then fill out all the appropriate paperwork and then interact with the insurance companies AI agent that could simplify a lot of things take a lot of the burden off of the the doctors in the medical system hopefully and have them just focus on the patient instead of on the massive amounts of people they have to do scenarios where a human talks the AI through moving through sort of this maze with different corridors and windows and doors and the AI builds up sort of a mental model a map of the area that is being described so we see that gpt4 accurately tracks all the locations of the room from the exploration and then visualize them correctly and here's an example where unlike the user assistant for humans this is kind of almost like the reverse it's where you have a human assistant as in you have a human being helping you as opposed to an AI helping a human so so gpt4 is given a two real world problems to solve and a given a human as a partner that is a humor is a very flexible agent with very little constraints so he can also respond in natural language would it be crazy if at some point during genre reviews you could sort of like promise the interviewer that you're able to get along and work with um AIS without any problems like you're oh I'm very flexible I'll do whatever it says and so both problems were real problems faced by authors of this paper who responded to gpt4 in such a way as to track the situations they faced and so the problem here was that the kitchen ceiling is dripping water that's that was the humans complaint and so as you can see here the computer starts by saying check to see if there's a bathroom or other water sources directly above the kitchen and then the guy goes yes so then he said we'll go upstairs check it out and um as the human being it's sort of like the the eyes and ears on the ground he's looking around he's going okay yeah there's water around the bathroom and the computer's instructing them okay you gotta check the seals you got to make sure that they're not worn or damaged and it's basically troubleshooting the issue right so it's walking him through how to troubleshoot it how to find the source of problem and then finally it's telling him how to fix the problem so if you think about it the only thing that the human being really has to do to kind of get this ball rolling and say hey there's a my kitchen's dripping water and the AI will basically I mean if he has the minimal intelligence needed to kind of follow those instructions the AI will walk him through to how to fix it I think the next level of that is sort of having an AI boss that is basically telling you exactly how to do your job and um it's closely observing everything you do to make sure that your outputs are correct so so I encourage anybody that's interested in this to really go through and check out some of the stuff some of it's pretty deep and pretty complex so depending on where you are in terms of computer science and math and a lot of the stuff this this might be out of your wheelhouse but a lot of it is very fascinating kind of shows where we where we are within Ai and how quickly it's advancing and just if you take kind of like this and where we were two years ago and you kind of project forward it gets really weird really fast because I feel like we're at the base of this mountain that we can't even see the top of but of course there's some people that are upset that don't trust this that are sort of um really against this stuff and you know what I mean some of the points that they present are valid I'm not being dismissive of it Gary Marcus is one person that um speaks out against some of this stuff and explains why we should be very careful with technology like this and why it's not quite ready for prime time and why maybe we should have some breaks and some sort of limitations that are in place before this thing gets out of control he's got a sub stack if you're interested um I've been reading for a while as sort of a way to see sort of the other side of the debate sort of the all the cons and all the dangers of just letting this AI sort of develop unbridled so you know some of the solid critical points does nobody else see the extreme irony of using something that produces false information to evaluate other false information to establish its veracity those are part of the paper where gbt4 asked to act as a judge to judge its own responses to see how well it could predict how a human would judge its responses so it's this this is kind of making fun of that in a way that it's like well if it produces false information why are we asking to judge how it produces false it's sort of a loop another criticism is there's a real problem here system researchers like me have no way to know what a bar gbt4 or Sydney are trained on companies refuse to say this matters because trading data is part of the core foundation on Which models are built science relies on transparency and so Gary Marcus is saying that you know here's the thing it's called Coca-Cola wants to keep their secrets that's fine it's not particularly in the public interest to know the exact formula but what if they suddenly introduce a new self-improving formula within principle potential to and democracy or give people potentially fatal medical advice or to seduce people into committing criminal acts at some point we would want public hearings Microsoft and openai are rolling out extraordinary powerful yet unreliable systems with multiple disclosed risks and no clear measure either of uh of either of their safety or how to constrain them by excluding the assigned to a community from any serious insight into the design and function of these models of these models Microsoft and openai are placing the public in a position in which these two companies alone are in a position to do anything about the risks to which they are exposing us all and he's saying that we must demand transparency if we don't get it we must contemplate shedding these projects down now it does seem that open AI started in in one place when Elon Musk and Sam Alton founded it it was supposed to be this open source transparent uh initiative to bring positive AI to the world that we all could sort of see and control and it wouldn't cause issues and somewhere along the line kind of like Microsoft got involved and now it's a little bit more opaque and uh we're not able to see all the data at the same time it looks like the progress that they're making is just increasing exponentially to where in some ways it seems that Google isn't even able to kind of match that speed or at least it sort of maybe is being a little bit more careful a little bit more sort of safety conscious in this race towards AGI so certainly the concerns are valid there's something to this idea that if we're not able to see it then could potentially develop into something extreme that damages us all Counterpoint to that is if it is getting this Advanced then does putting it out there to where all the other nations potentially hostile Nations to us where they they could see it could that cause issues I don't know if there's a simple answer to this but one thing is getting to be pretty obvious it seems to me that this is an exponentially growing technology that is just beginning to get really powerful and the effects of this are going to be worldwide and are going to be very strong they're going to affect each and every single one of us I'm curious to know what you think do you think this is something are we summoning a demon that's going to come out and just kill us all and make the world a much worse place or do you think this is going to be an amazing thing that's going to allow everybody to work less and just enjoy more time and not have tutorial day in day out or do you think it's going to be somewhere in between do you think that perhaps this is just a passing fad I'd love to know more let me know in the comments I read every single one if you want to keep up to date with the latest AI news explain simply not necessarily the theory but more about how it's going to be affecting the regular people and all of us sign up for my newsletter I'll leave a link in the description it's natural 20. thank you for watching my name is Wes Roth I'll talk to you later