The $1,000,000 problem AI can't solve

Key Takeaways

today I'm going to talk about I'm going to show you the $1 million problem that AI cannot solve and the reason I think this is relevant is number one because I think any human watching this video right now is capable of doing much better at this problem that any AI any machine learning model that we've built so far just so you know the current leaderboard people submitting Sol Solutions uh to this problem the top of the leaderboard the top solution the best one is only capable of solving 38% of the tasks that it's in 38% any human watching this video I'm sure is capable of doing way above 38% the second reason this is relevant is because we keep listening to people we keep hearing reports that AI is going to take our jobs in 5 years that in 3 years from today they're not going to be software developers and I honestly think all of that is I honestly think in five years we're going to have way more developers than what we have today I think we're going to produce way more software than what we've ever produced before so we're going to need more people but especially I think that models unless there is a is a breakthrough there is a breakthrough that we have not had so so far models today are fundamentally stupid they're very helpful they're very useful but they cannot produce novel solutions to problems they have not encountered before now how how do you understand this because everyone watching this video I'm sure you've seen these models working and they're certainly helpful they're certainly impressive but keep in mind they're only capable of generalizing within the problems they've seen before there it's just a big interpolation machine with a huge database of problems and things they've seen and they're capable of mix and matching those Solutions they've seen find the patterns mix and match them to produce new solutions to problems and that is definitely very good and I'm happy we have those but these models do not reason just like humans do these models are not capable of just using a few building blocks to find novel solutions to problems they have not seen before and this specific problem that I'm going to show you right now proves that AI is not good at solving this so let me show you my screen and I want you to go to the arc price.org website so here is the URL so the arc uh Benchmark is basically a set of very simple problems and we're going to go through one of those right now that AI needs to solve so here is what a problem looks like they're going to give you an example input which is this this little grid here in this case is a 7 by seven grid and this input shows some squares with colors and they give you what the corresponding output will look like okay so in this particular example they give you these three uh blue squares here and then this is the corresponding output now notice the pattern here for this example the solution seems to be adding a blue dark blue color uh in this space just to complete this Square let's look at the second example uh so you get the same idea but now you get three of these shapes and exactly what I told you they added the solution is adding this blue dark blue uh spot right here to complete the square so now you have to solve it now the solution that you need to provide uh they give you a new input and now you need to provide a solution so something that you can do here in this website is I'm going to copy from the input I'm going to copy that from the output and now we need dark blue and I'm going to say this is the solution okay just very simple I'm going to click submit the solution and this is correct okay so that was an easy problem notice that the knowledge that I need of that unit to solve these problems is very basic I need some idea of the concept of colors uh positioning of shapes understanding how to find a pattern is a very basic uh very basic building blocks so let's look at the next one because all of them are going to be different so in this case look at what we got we got a red dot and then next to the Red Dot we get four yellow dots okay here is a red dot we get four yellow dots here is a red dot we get a four yellow dots okay so that seems like a pattern let's copy this from input and let's do the yellow dots right here okay okay now the blue one it gives me these orange dots next to it in a different shape here we get a blue one here we get the orange ones okay do we have a blue one here yes oh wrong color you also need to pick the same color obviously okay so what about this magenta one well here let's see ah there is a third there is a third example here great and the magenta one is left alone see the magenta Genta one when it comes in the input is left alone in the output so yeah oh no there is one here I almost missed this you know light blue color I have it here and it looks like it's left alone as well so I'm going to submit this and see what happens boom correct so those are those are two out of two and by the way you can solve a bunch of them you get six here there is somewhere else here in the page where you can solve uh the basically the test set that they offer the public test set and some of them are harder than these two these two are pretty simple but I want you to get the idea that any human looking at these problems is able to solve them I will even say that small children will be able to solve with basic instructions and after seeing a bunch of these problems maybe four or five of these problems they're going to pick it up and they're going to be able to solve these test cases but the models today let's go to the leaderboard the best model today the best solution not model but the best solution today is at 39% only 39% of the problems this mind's AI has been able to solve with their solution the price here will be released when a solution gets to 85% now why are these models that seem incredibly helpful so dumb at solving this simple Arc problem there is one very simple reason and is that the llms out there have not been trained on this data have not seen any anything like it these problems are very easy to reason through for any human that's capable of solving novelty anything new we're capable of adapting you guys are watching this video right now that you've never seen me before you know exactly how to react if I remove my hat you know how to deal with situations that you have not been exposed to but these models aren't they aren't able to do that so now we present these models simple problems like AR and these models show their true colors they cannot deal with uncertainty they don't have the basic building blocks to plan and to synthesiz a problem or a solution for a problem they have not seen that is the reality of it that is the reason I I do not think this problem will be taken care of or replacing software developers anytime soon sure they're going to help software developers sure they're going to take over a bunch of mundane tasks that we don't want to do but they won't be able to solve software just like a person will at least not without new breakthroughs that we have not found yet so we are not there we might find all of those breakthroughs we might create something tomorrow and I'm going to have to eat my words but so far with the knowledge we have today these models cannot go beyond that let me give you another example I've been coding em Mojo lately Mojo is a brand new programming language is not even done yet okay and I'm using co-pilot to do that which is powered by an llm a large language model and it sucks big time copilot is constantly confused because it does not understand how the language works it doesn't matter how big my file is copilot keeps getting confused and keeps offering me suggestions either in Python because most of module looks like python or JavaScript because it detects that I'm typing things that are not python like copilot is not able to adapt to module Co pilot is not learning on the Fly because these models are not capable of doing that so here is one recommendation uh I suggest you go to the play section here it's super fun you go to the play section and in the play section you can pick uh hard problems and easy problems that are from the public evaluation set on the training set okay um these are cool because they get a little bit more Lex like in this case here you get an input that is a 2X two square like you get basically two examples but the output is 6x6 okay so now you have to determine what the output will be you have to resize it and it's harder to find out the pattern harder to find the pattern but for you human being this is very simple actually after you do two or three of them I promise you're going to score very very high uh something else that I wanted to mention here uh here uh if you go to the technical guide you're going to find how these problems are represented internally and surprise surprise all of these problems are just sequence of numbers like this is how the llm uh sees this problem I mentioned this because some people have told me well but now that we're building multi modal models now that we're adding Vision capabilities the llm is going to get better we don't need Vision compability to do this internally the way the problem is represented it looks like this right so this is the train set with three examples here here is a test set with one example here and as you can see there just sequence of numbers where the one represents an arbitrary color like what you see is just a simpler way for us humans to see but we can train the llm or we can teach the llm or give these problems to the llm directly as a sequence of numbers and llms are sequin processing machines they are supposed to be very good at processing these sequences and yet they suck at solving or coming up with solutions to these problems okay so again it's a fantastic way to try to understand how these uh machines to call them somehow work internally and to realize that they are not reasoning they fake it very very very well they are extremely helpful they're going to provide a ton of economic value but they're not going to take over what we're doing anytime soon because they cannot think so hopefully you get value out of these subscribe if you did like this video and I'll see you in the next one bye-bye