More Chessboard Computer Vision AI - Data Science Uncut - Sep 13

Key Takeaways

hello everyone late night start a little later than normal but i hope you all are doing well it is what day is it some september 3rd 2022 it's a tuesday i believe um and i'm streaming here out of home base i'm thinking we're gonna try some um doing some computer vision stuff like before uh with chess boards let's see how far we get i haven't gotten very far so far so uh continuing on that trend potentially but i hope you're having a great night we can just chat and hang out it's gonna be probably a pretty short stream um i was thinking of trying to make this as simple as possible and see if we can just like do the bare minimum when it comes to computer vision classification so let's see what we got here um obviously um i'm not prepared and i don't have anything loaded up so that's just how i do things but we're gonna go ahead and do this switch this hopefully you can see that terminal prompt if you're in chat let me know let me load up my um channel just to make sure i can see if there's anyone there watching and there is not so it's just us by us i mean me there we go all right let's make this a little bit bigger and let's show you what we have here i have a chessboard on this camera and i thought it would be interesting to see if we could do the bare minimum and maybe work through what the labeling is going to look like on this labeling system and potentially what the um scoring metric could be for how well we would say our computer vision classification is now one thing i will mention the camera is really wobbly it's attached to my desk and i have shaky knee syndrome um also known as add so it might you might be shaking a little bit i apologize for that hey david jackson welcome to the chat you're the first one here so glad you're here um not much is going on we're gonna see if we can uh make some make a data set and talk through maybe how we're going to uh label this so we need to take things step by step this is my pizza pizza pizzas which i've typed up every time a new subscriber comes in i spin a wheel and one of the things i have to do sometimes is uh type pizza 100 times so that's why p you saw pizza there but we're going to think through two things number one is how do we um how do we set up the output or the labels classes all right so basically if we're going to make a computer vision system that's capable of identifying the position of the board we're gonna make some concessions we're gonna we're gonna start out simple here and we're gonna assume maybe that the a1 corner is always gonna be here in the bottom left bottom right sorry from that viewpoint from where i'm sitting it looks like the left but from your angle it's the bottom right and then we so that means that white is always going to be starting on this side and we don't have to worry about potentially the camera being like from behind white's position or from the other side of the board we're just gonna make things simple by the way david j let me know how you're doing tonight too i'm interested to know what do you what have you been working on you're my one chat companion so how do we set up these labels um i guess it's gonna have to be like a 64 i mean we could do we could do it eight by eight tensor but i guess it doesn't really matter we'll just flatten it to 64 tensor and we'll start with a1 and then we'll move our way so we'll start we'll go like in this order b1 c1 etc so that's gonna be starting here in this corner so when we write our um ground truth labels we'll have hold on let me change these settings [Music] i'm looking at big foot sighting data that sounds amazing can you send me a link i'd be interested in taking a look at that bigfoot sighting data that sounds like a lot of fun all right so back to this a1 here will be where the first thing we'll label we'll go all the way through to a2 a b2 etc and then we'll get to the final row which will be will end on f8 g8 h8 right so for each of these values in our tensor hey what's up lemon skate i hope you're doing well tonight we're just uh trying to figure out how to set up this labeling scheme in our computer vision system so then we're also going to need some classes right now i do want to just leverage the fact that we already have a data set that's sort of done this um where they've labeled hey we got gustavo in the house welcome how's it going um so there's this chess data set that someone has created already and we should maybe use the same labeling technique that they use so we did train a model using yolo v5 pi torch i think um i think we have this data on my computer so let's go into our chessboard vision repo actually i think it was in yolo v5 data maybe it wasn't that maybe it was in um chessboard vision i think i see it there this chess image data set so let's open this is this the same data set let's read the readme actually this is the one that uses blender chess image data set now i'm going to get distracted and start looking this one up this is like a 3d rendering software to actually get the example images of this of a chessboard [Music] what's this chess man data set oh we've seen this before this is just like a bunch of pictures of different uh chess pieces for oh we could use that for something maybe later here you go what do you what are you pointing me to david jay oh this is your tidy tuesday um [Music] where's the bigfoot sighting data get the data tt load oh so this is like a tidy tuesday they just kind of link it through their module so you can't download it directly or you could just download it directly like this bigfoot.csv i should put this on kaggle if it's not already on there because that's kind of cool [Music] here's the csv let's view the raw date whoa that's a lot of text for for finding bigfoot are these like full articles um yeah that's a cool data set i'm going to check that out later data set of rendered chess game set images this is cool so this is another idea that i've probably gonna get sidetracked on um oh look there's a paper that came out about this determining chess game state from an image let's see how these people did it maybe copy their setup the problem of recovering the configuration of a chess pieces from an image of physical chessboard is often referred to as chest recognition application span from chess robots remove recording software a digitizing chest positions images particularly compelling application arises in amateur chests where the casual over-the-board game may reach interesting position that players may afterwards want to analyze on a computer that's kind of the position we're in we're trying to take this board and have something that will detect the position to this end we put forth a new synthesized data set comprising of rendered chessboard images with different chest positions camera angles lighting setups furthermore we present chess recognition system consisting of three main steps board localization occupancy occupancy classification so that's which uh piece is occupying each position in the board and piece classification so that makes sense the two ladder steps we employ two convolutional neural nets to make the traditional computer vision techniques for the board let's see what their accuracy are do they say it here however chess sets vary in appearance by exploiting the geometric nature of the chessboard the board localization algorithm is robust enough to reliably recognize the corner points of different chess sets without modification using this algorithm in conjunction with careful data augmentation we can extract sufficient samples let's see if we can get this paper to work look they have even they have their own the uh repo for it three methods of installing running chess sog using poetry i don't have poetry uh using pip and using docker let's also see in this paper what accuracy they're measuring this by so this is kind of getting at the root of our problem um yet the approach they're talking this is talking about um previous work and they're saying let's just skip to the end the approach that the last people they mentioned was captured in depth camera who who represents chessboard recognition system using basic neural network that is not convolutional oh that's interesting you like p pep hater i do as well so this achieves an accuracy of 72 percent let's see what did they say what their accuracy is i'm surprised that it's a lot of wooden chess boards all right original end image detect edges we've done this before we've done this before then this looks like it's from the top down no this is using augmentation this is using augmentation to identify these center points and then to augment the data to make it look more like um a top down board which is interesting because then it kind of modifies it for the fact that there might be um different angles that your camera is looking at right hey herbie hoover welcome to the chat how are you doing how is everyone hey by the way big announcement coming later this week later this week check out for my channel i'm going to be releasing uh um something new we're gonna we're gonna have a something exciting announced here so make sure you stick around they trade a total of six cnns with 12 output units in the final layer for piece classification okay so 12 output units in the final layer oh that's in piece classification so they are doing this in steps it's not an end-to-end approach the first one is identifying the board then they kind of isolate to the board they do board localization piece classification and let's see what their wow their validation accuracy is 99 on the peace classification and let's look at their confusion matrix this is a great well written uh paper good evening data basics how's it going herbie hoover doing well had a recruiter reach out to me for data science world nice i love it is it a position you're excited about i hope so so what's the confusion usually i guess this blank means that thinks it's blank so a lot of times it thinks upon so the most confused is between a black pawn and thinking it's blank um could it be because of the background we were having that issue too inference benchmarks um so it takes about 0.3 seconds 0.35 seconds for a gpu classification and 2 seconds for cpu nice and they have a github repo [Music] so should i see what the requirements are on this [Music] so i want to also see their model like their output because that's kind of what we're trying to get at utils config chest sog so corner detection occupancy classifier um they are using pi torch so this is a convolutional neural network and their last output layer is 256 to the number of classes okay so this is just what's the number of classes gonna be numb classes gnome classes is two oh so this is just identifying if it's occupied at all let's read about the occupancy classification to make sure i understand this we find that performing a police identification directly after detecting the four corner points with no intermediate step yields a large number of false positives i.e empty squares being classified as containing chess pieces see figure five yeah that's what we were seeing in the oh wait this is figure five looks like this oh so it identifies this square and thinks that there's a piece in it interesting um large horizontal gradient intensifies give rise to vertical lines oh wait well okay to solve this problem we first train a binary classifier to identify so that's why it's zero and two um but this runs on each of the squares independently it looks like let's see what their evaluation function looks like hey herbert welcome what's up we're trying to do chess classification chessboard classification and i'm reading up on this paper that looks really nice um so this module that brings together the whole recognition pipeline into a single class so it can be conveniently executed awesome okay so let's let's go ahead and install this let's go ahead and install this so let's conda activate uh conda emv list [Music] it's kind of awesome but also kind of annoying that the problem i wanted to work on is already solved um but it can be improved conda activate chess viz um we can at least see how well it does hey we got a chess ch chessiest w guy following thank you for following so we are going to go into this repos directory actually we already um activated that so let's end this and jupiter lab this and also in our chest viz repo uh conda environment we are going to do some pip installing hopefully it won't mess up other things that we've installed but we'll we'll see so we have to clone this [Music] chess og cd chess cd chess og [Music] chess cog chess cog uh hello we're doing chessboard recognition stuff yes chessy ttw we are we're trying to you can see here i got this camera set up on the board and we're gonna see how well this uh this pipeline that someone else has created as their master's thesis performs so let's go into chess ogg and we're going to pip install this pip install this we're in our chest viz library or a con environment and then we're going to see how well it runs just on this uh chessy is ttw how did you find the channel i'm in interested to know is this supposed to rest recognize the piece also or just the board oh the the piece the pieces i just don't have pieces on it yet so i was gonna break some of those out here while this is installing it's probably a good time to do this let's just let lady luck decide what comes out oh it's a black pawn so we'll we'll see how well it does with that actually it was saying that the identifying a black pawn gets mistaken uh for an empty square at most twitch algorithm probably i want chess and computer stuff oh you're in the right spot yeah you're definitely in the right spot um so i just installed that let's clear this workspace this is my youtube stats not what i want right now let's start a new notebook called chess og um testing out the chess cog cog right chess cog library let's put the reference in here to chess cog and then let's follow their directions we can download the data set and models okay so we need it [Music] first we need a download data set okay so there's there's like a um download data set function to run we kind of just want the models yeah this is four gigs so we're not going to download that let's just try to download the models to start i'll download the data set later um are you trying going to train it on a specific piece that you have yeah so we've done that before we've done that we trained a yolo model on the these types of pieces and we had pretty good success with it um not perfect but pretty good [Music] so download it to attempt directory and then i put it in media chess models okay so i put it in the models directory so now if i look here there is a models directory and this is the occupancy classifier it looks like it's a that's the resnet um based classifier why's have a train in val folder oh that's just like the training logs uh for tensorflow tensorflow um sensor board sorry then they use inception v3 as the what's that the piece classifier all right so just a quick overview of some machine learning models if you look up tim library not tim like a person named tim but the model pi torch library tim it's like the ultimate library for computer vision pre-trained models hey klipt hey i recently started watching your videos thanks for the help on data analysis and cleaning that's from nagas nagasa from youtube thank you for watching them i appreciate it so um just so everyone's on the same page pi uh tim i'm showing you this because ross whitman is a great guy who maintains this um this library that allows you to easily pull in a lot of pre-trained models why do i why am i bringing us here why am i bringing this up because i want to show you some of the main models that are used these days uh create model does he have them listed i thought he had them listed somewhere overview list models and pre-trained weights yeah so there's a lot more than just these but this is just lins like listing the dense net models yes actually let me see if i think i've tim installed here import tim all right so it looks like we can do list models and this will lens list all the dense net models but let's just do star in here will this be crazy all models look at all these models we have to choose from so some big ones that i've used in past competitions so these tf versions are just like the tensorflow versions of the same models but let's just pretend that those don't exist those are like tensorflow efficient net but efficient nets big ones to go through at least like a year ago a year and a half ago for all image classification stuff like this was the go to efficient net always use though they train quickly they're pretty reliable now a little bit older i think are the resnet so um obviously it depends on what you're trying to train a model for but resnets are pretty handy you got a bunch of that and uh then they're like these fancier resnet like sc resnet there's um uh res next where are those res next that are popular and then these different like the reason why you see a bunch of different ones is because they have different archite like the same underlying architecture or idea of these models are the same but like for efficient net at least these are the size of the model and you might think to yourself oh well i want the most accurate model possible so i just need to get the biggest one like a b8 um the problem with that is number one take forever to train number two if your data set's not large enough or your the image size is not big enough it really doesn't give you that much of a improvement if anything it might overfit too much and number three is for inference time if you're using a large model it's going to be super slow so that's why you don't necessarily jump to these larger models all this is to say it looks like the backbone that they're using so backbones like you usually take these models and then you like slap on whatever um output you want to be at the end of this model and that'll kind of let you predict whatever like if you're doing a binary classification it might just be a one class and if you're using like default image net what 10 24 or however many there are in there hey there big fan so working a little bit out and wanted what are you saying rohan hey there big fan still working so i'll be in and out want to thank you for all your great work thank you for watching i appreciate it uh just autumn said the cnn's the current state of art is covnet okay so covenant's the best now covenant tiny is probably a good choice for basic applications good to know good to know so we can see like granite all i've learned about why are covenants not in here most of what i've learned about image models is from kaggle competitions and typically in a kaggle competition whatever the best model is is going to be used because the people know how to win right um need the latest version of tim likely okay so i just need to upgrade my tim i have 0.3.4 what's the latest version yeah so i haven't done an image competition in about a year so if cub nets better i'll give that a try but um there are also like specific models for classif for like segmentation that don't necessarily fall into the same category and then you have these like uh vit type of they're like transformer type of models that are supposed to be good haven't used them too much in exception i've heard of used a good bit um yeah so these are all the like models that are out there but that's not the ones that they're using so let's go ahead and use their documentation oop why does this look so huge all right so this says chess.og recognition.recognition path to image um hey ada weave welcome to the family i'm so glad you're here with us tonight um and this is how their output looks like that's cool so it outputs it like a chessboard let's do let's see if i i don't want to run it from the command line like this is a module i want to run it in my notebook so the way i would do that is um from chessugg just cog keep on wanting to call it chessog dot let's just see if i can import oh i'm not in the right environment gotta go to chess chessboardvision uh i can just should i create a new environment no i just need to make chess viz accessible in my in my jupiter lab so make con environment found in jupiter i go to this site so many times just to run this one command line so um pip install ipi kernel what is it ip kernel python dash m ipy kernel install user name and this is going to call be called chessviz now that now this should be available in my jupiter environment as something i can choose let's see what we got here uh chest fizz let's go let's go people let's go hey wolf goes nice welcome all right let's delete this stuff let's import let's see if this works i progress not phoned um maybe i should pip install yeah thanks for joining the family wolf goes naya let me know how you found the channel in chat that would be great requirement already satisfied so do i need to install eye progress that's installed now okay user install upgrade my widgets so maybe i need to install ipi widgets let's go back import this chess all cog all right so then we're gonna do um let's see what recognition has let's see in recognition recognition if we run this as a module it would run this main which would parse out these arguments read in the image convert it from bgr to rgb okay that's okay then it makes this chess recognizer so that's kind of what we want that's what that's what we want to pull out of here it's the chess recognizer and let's see where the chess recognizer it's it's a class that's defined in this uh file so let's do uh from chess cog recognition import what was it called chess recognizer okay so now we have a chest recognizer object that we can use the same way let's go ahead and take this camera and close it here so that i can uh also import cv2 cv2 uh video capture and then i need to figure out what device this is i'm pretty sure it's device seven and let's read this that should just read the current image from that camera and we can check that out by also importing matplotlib pi plot as plt we can do plt i am show this image there it is it's the wrong camera that's my secondary that's my like old school camera so let's uh do cap release this just so it's released now the light on that has turned off and see if it's five there we go and you might be saying why is this image look weird that's because it's in bgr instead of rgb and by flipping around rgb the the third channel we can make it bigger so let's make this fig size bigger okay there we go any questions anyone no questions okay so now we have a we have a piece here on in the middle of the board and we're gonna see how well this image recognizer works thank you for joining the family you're part of the family i appreciate it if you're watching over on youtube you can come over on twitch if you're on twitch just stay here that's the main chat but um really you can watch from wherever you want and hopefully i won't get a copyright strike like i do every time that i stream to youtube because i swear i'm using uh what says it is a non-copyright music soundtrack in the background i'm trying a different one today but um it always still gets flagged okay so we have our image we've converted the color or we can convert the color like this to basically let's just copy everything from here and see what they're doing um so we don't want to read the image because we already have it convert it to the correct coloring create our chess recognizer now we need our classifiers folder hey grum panera soros welcome to the family you're part of the family now and also you wiggis bieber how did you guys find the channel let me know in chat we're creating a chessboard live detector so let's go up this so i don't know what the recognizer sends back and i don't know what the classifier folder should be i think it's going to be repos i don't know if it takes a string directory um but let's try this chess cog models maybe just the base directory will work unsupported operand for for string and string okay so let's see it's probably going to take like a a path object classifiers folder is yeah the path [Music] it's uri from recap import uri what's recap what uri is that because it's trying to get a url okay so args.color let's see what this prediction method takes so it takes in the image so we've we've created our recognizer it's working correctly now that we think this takes in the rgb image which we should have now actually i ran this twice so the image probably is not the right color it's converted back just that's a consequence of me converting it twice so let's actually read the current state of the board then it's going to be converted into rgb we have our models folder and we're trying to predict now i didn't give it anything for the turn okay so that's the current player return to the board look at that hi medallion stuff is looking very interesting question did you create a new jupiter kernel with called chess viz to have it dedicated to chessog but how did chessog in computer it's not in pi pi oh so i just followed the directions you can pip install you can pip install uh python library without it being in pi pi so you the way you do it is you clone the directory and i do this from my own packages from time to time time to time you clone this i don't know why they have a space here this should not have a space and then you cd into the directory and now they have a setup.pie in here or it looks like not they have this poetry uh which is like the cooler way of managing the packages but they have it set up in here that you can just do pip install uh period when you're in this directory and it'll install that package um so let's uh let's do two things at once let's plot this image can we crop it a little bit from like 100 to 600 and then from this from like 100 to about 600 now we're a little bit more cropped in on the board it's still got that angle going on but let's um wow i didn't know that this package already existed this is pretty awesome so now that we have this let's try making a little bit more complicated of um of a board setup what is anyone in chat it's not correct look at the look at the color where the pawn is you're right so is this assuming a1 is always the bottom left corner okay uh good good observation grump i'm so glad you're part of our family all right let's test this out by um i know you can't see this camera but i'm going to move it to the a1 square we are going to re-run this uh clearly it didn't read in the latest image hey hi i'm kyle welcome to the family i hope you're doing well how's it going i um i moved this why isn't this showing it and this cap.read should read from this device so let's release this device just to be sure cap.release and do this and see where it is okay so it's in a1 now now it's actually an a1 let's see what the board says the board doesn't know okay so this is ferret this is fair the board doesn't know where the correct position is it doesn't know the position of the board granted like there are light and dark squares but it's just assuming that the bottom right is is a light square let's try to see or you can just rotate the image yeah i can or i can just physically rotate this board and and drop everything on the ground i wish you could see this camera live but then i won't be able to create the capture device and stuff um so we're gonna cap that release not sure why i have to release each time it should it should be fun so now the bottom right square is corr it thinks it's a night i didn't even realize that um so not perfect that's good that means that we can improve upon it i'm surprised it thinks it's a night though yeah we want our solution to work really well on um let's also crop this a little bit more actually no we're good for now um we want our solution to work pretty well wasn't it showing upon earlier yeah i guess it doesn't like the fact that this pawn is down here let's like maybe put it up here on i've just moved it to h4 and let's see how the the classifier looks like so now it still thinks it's a pawn it could be do something with it something with like the angles yeah exactly high mq it's like the angle of of the position all right so let's make it a little bit crazier does anyone have a favorite like chess game they want me to set up i actually want a1 to be maybe rotate this uh pawn on h1 is illegal position does it consider that pawning h1 is a black pawn on h1 now that's a legal position fischer game six yeah we can do that um wait upon an h1 is legal for a black pawn it would just be promoted to a different piece so oh yeah so i guess then technically um it wouldn't be okay so let's do look up um fisher versus spaz game six i think i have this printed out like a visualization of this printed out as some artwork somewhere else in my house [Music] let's find the key moments i don't know what the key is moments the key move of the game is the move 20 e4 e4 so i'm gonna set this up i'm gonna set this up bear with me uh pawns can't be on rank one or eight true i was forgetting the fact that if they get to the eighth or first rank then they're promoted um fair though fair fair point let's also let's delete this move this down here uh let's do some let's do some cleaning up so let's do let's create a get image so this should return our image and the board and i don't want to do this and then we'll also do the releasing of this capture device down below here so it's like cap release here remove it from here uh so what are we doing take our capture device let's just make this variable cap devices five yeah captive ice because otherwise might get confused with like gpu device hey why not release first if we don't have the object um then we can't release it hey anna yamani welcome to the family i'm so happy to have you here we are streaming about uh chessboard image recognition um i'm also gonna make a function called plot image and we're gonna take in the image i do this a lot like it's kind of annoying just make the same function i think this is axis off and then this should just help us to run it easily so now we should get our image and our board if we run this get image class uh this should not take in an image it's just taken in a capture device um this is not going to work because we need to release this first and now we have our image in our board and let's try our plot image on this image there we go it's kind of big maybe too big that's what she said um then we're gonna go so make this a little bit smaller there we go well it's like when i make the font smaller it actually makes it small bigger okay so there we go uh you should only do the image capture once not inside get image [Music] think of a lazy global variable initiation once you should be able to forget about the release so you're saying that the first of cell of line 46 well it's not cell 46 anymore now it's 52. uh but wolf goes nice and saying should only do image capture once you're saying okay so you're going to create this capture device and then only pull in the image once like do all this once the cap cap.read or feed in this capture device in once because we could do we could do this the reason why i'm doing it this way is because i want to every time i run this i want it to pull the current image from my webcam like i could make this like this remove the release uh this cap device is gonna be five i guess it wouldn't need this capture device anymore all right i'll humor you so then i need to release it here uh get image class oh that that's not supposed to be in here that needs to take this and of course we need it we're in this infinite releasing state but there we there we have a we have the position we have the plot and we have the board which has e4 uh that's h4 okay so let's let's make let's get this set up the same way that it has it here i'm gonna grab myself crazy it's like this in real life irl now it's in the top right which is e4 e4 light square e4 oh sorry e8 i'm going crazy it should be like this can i even think so a1 should be in the bottom left which it is why is this not right then because this is an old picture that's why we have something strange going on here what i'm looking at here is different okay what i'm looking at here now is what i think it should be there we go now we have the right device uh this image is too big to actually see though let's make this smaller stack not empty tensor list there we go okay so we have some cropping issues that's because i've been finagling this board around i moved it uh too far in one direction because i really like the band one direction so let's go ahead and do this took out the crop completely um it needs to move a little bit this way fair enough run this again there we go and let's see what it says this position is it says it's h4 great so zero don't call this again yeah i see what you're saying wolf goes nice so you're saying that i should create this this first and then in theory i should be able to run this multiple times every time it will capture the new image right it was being strange though and not actually working i'm going to move the pawn to so you can't tell because it hasn't moved i'm going to move the pawn to e4 and smite smack dab in the middle of the board let's see if the image actually updates it does not see usually it works fine it could be because it likes maybe i just need to restart this notebook yeah that's that's why i was doing it this way um but maybe now that restart the notebook it's gonna be better we'll we'll get this together all right so now you can see it's on e4 printing the board state it's on e4 try again e4 oh wait i haven't moved anything so we can't tell i'm going to put the pawn on c2 [Music] so we get this cap video io issue could it also be because i have i think i closed my yeah so i'm just gonna go ahead and release this each time because i know that then it works now it's gonna work now i'm gonna move it over here [Music] it shows it in the new position and it shows the piece has moved to f4 great now let's set up a full position we've only messed around with the black pawns now we are going to set up this position and see how it goes so i'm going to find some pieces you guys just chill out here for a second um no wait let's let's get up this other camera see if it'll actually work i need to release it i need a release release the stress ah not that uh cap that release just so you know if you if you haven't worked with this board object before we've done this a lot on stream but this is like uh the type of object this is is from the chess library it's really handy because if you actually print it it'll print like this like it shows where the pond's position is i think we can get the fen i don't know why it's doing that weird thing but you can do a lot of stuff to this like um uh validate that it's a um correct position speaking of which let's let's check to see if i can get this pawn on to um g1 and see if it works or if it says that there's a pawn on g1 or if it's gonna fail because it knows that's an invalid position no so it was i think it was just a classification issue um but we can also like just like see where how many bishops are on the board i mean there's can claim draw board fen this is what we used a lot before so this is like the fen uh notation of a chessboard if you don't know about that just look up wikipedia fen uh we wrote a little function to convert between the two um and i am going to i now have released that let's see if i can actually open up my cheeser nope and show you guys me setting up this position so let's get this over here get here so it's not too boring maybe you could show the info one plot yeah the thing is that this what it's showing here the second board is it's not an image and we worked on this before it's actually like an html rendering or a svg rendering of the board state so um so if we go to like the python chess library let's see if there's um yeah i'm not gonna get sidetracked on that too much but i i totally get you ala i i've tried that before and i think one of the solutions was just to save it as like a png file on on disk and then read it back in anyways let's see how well it does with a legit position what let's vote in chat let's have a vote by the way i've been ignoring the youtube chat so if people are no no one's talking in that let's do a poll how many pieces will it get correct all uh 50 greater than equal to 50 less than 50 or none i'm gonna make a five minute poll while i start this up please vote in the poll let me know what you think is going to happen as we see as we do this so just to show what's going on here we have the poll we also have the position that i'm hoping to set up here going on from the famous game and then we're going to use the detection algorithm to see how well it can pick it up can you guys see all this stuff does immunization impacts on classification immunation what sorry what does that mean um so i probably should have had all the pieces out first i'm gonna need some ponds i'm gonna need a white king get them all on my lap probably drop them somewhere um all right let's get some pawns h2 oh yeah so this is like this is gonna be the white position eight two h2 uh tell me if i'm messing this up by the way let's make this smaller scold me i should have given myself less than or more than five minutes to do this so i'm super slow uh put this like here-ish nothing's working there we go all right we got this nope we got this on h6 let's just go with all the white pawns first white pawn here we got a queen we'll have a wipe on here moved to e4 [Music] white squared bishop we got two rooks out now for the black pieces let's work our way from the top to the bottom we obviously need to have a black king now it's going to be interesting with this one because this black king is kind of like hanging over the edge you could see the background of my flooring i'm sure that's going to screw some stuff up let's have it here let's put a rook here and a knight here on on d7 [Music] please tell me if i'm screwing things up hi what are we doing today hey welcome how's it going whim we are using um algorithm that detects the position of a chess board using images we're using the goal is to create one that we can um that we can design ourselves but we're kind of using this paper that was released that um they released their code that does this we're seeing how well it does and maybe maybe it's so good we should just give up on this project and do a different one but you know i think it's maybe a good like kaggle competition sort of idea um do we have the position correct i know you can't see all the way on the left side of the board the votes are coming in people think that greater than 50 percent um is going to win some someone thinks less than 50 will be classified correctly and someone says all i actually my opinion is i think that the um oh white king yeah wait we need a white king hey speaking of monarchies no i'm not going to go there [Music] it is very relevant right now so we have the position set up here have what we know to be the true board position that i'm trying to get that's kind of out of frame here i'm realizing that you guys probably can't see this and then we have our voting i'm actually going to vote for less than 50 just to like just to be crazy here let's see how well it does so we have the position set up our polls about to end please get your vote in now it's your last chance before the doors close greater than 50 but not all i guess greater than 50 i didn't i wasn't explicit about it not also including all but um this is not not inclusive of all so let's see how well this does i need to close down this webcam version so that we actually pull in our capture device and this is the position now i'm not doing any crops to this we can try it with cropping out the the backgrounds we can also i can maybe put something on the ground to make it uh better to distinguish between these uh but let's do let's see what the board looks like who's the winner whoa okay so all is not correct all is definitely not correct um what are we what are we missing here folks we got some we got some misclassification with the king and the queen the black king may be hard yeah the white pieces are it's just totally missing this pawn why is it missing oh is it because it's um it's blocked i think we can see enough of this pawn that it should be able to pick it up that's just me illumination probably i got a lamp i can put a lamp on it they like the horse's most powerful piece yeah it's like hey let's just assume everything is a horse so it thinks this rook this rook is a knight it's kind of far out there to give it the benefit of the doubt um so the question is how do we score this how do we score how well this did so i i think i can pull in the fen position from this will let me just download this any of you chess.comers know this [Music] right no no not right click hey welcome to the fan family the architect you are part of our family now and i love it and i love you and thanks for joining us we're doing some chess board vision detection and all that stuff so here is the fpn position let's copy this and hey we got broom spoon welcome to the family you are my body import chess and then we're going to take a chess board object and we're going to feed it this fen i think this should work so this is the correct position this is the predicted predict uh prediction yes how do we score how do we score how did it get this the queen and the king mixed up maybe you should consider in some way the presence of or not of a peace in the position and the correctness of the peace detection yeah some sort of like um awareness of well probably what would be best is some sort of awareness of previous moves and then like if this is if we're playing a game then it should detect each move based on the change now i've watched a lot of these videos especially like just on my own for fun but i'll also watch a lot of these videos that we were trying to extract out the the positions for which actually we should maybe try to do that um and notice that there are a lot of issues with that too maybe the model you are using is based off pieces that that look different so the way yeah so we could retrain a model to specifically detect for these types of positions now this is the re repo i'm going to put in chat for everyone this is the repo that we used for um for using this detection obviously we want to like improve on this and make it better the way they created their images was through um through a what through uh a blender like 3d rendering software which i believe created like a bunch of different uh data sets speaking of which i i'll download that later um but they all aren't necessarily these chessboard versions but we could i think like focusing this to like the scholastic set these types of sets might be the best way to start and seeing if we can get an improved version of this but before we do that we need a score so i saw someone said hamming distance the architect can you explain more on hamming distance how we would label it because the way i was thinking of like the easiest way uh would just be like uh [Music] it's either a true positive like basically for each square it's either a true positive a false positive you know all that stuff so let's let's say this is the ground truth board and let's um recreate this board maybe before we do this we could try to improve this a little bit if you're doing from video you know the starting position of pieces so you can filter output logits and valid moves from previous state is that the plan that's part of the idea yeah yeah the architect um but i think the way we would score it is throughout the game so the there's two different things going on we have like the how well or what sort of architecture do we think could improve upon this classification but before we get ahead of ourselves probably the most important part of any good data science machine learning [Music] project is defining our metric how do we define what is good what is a good classification what is bad because this is not this is not good to have a queen be mislabeled or king being mislabeled as a queen and a queen mislabeled as a knight we know this is not good this is not ideal um but what if like this queen was correctly a king but this this knight was wrong or something else was wrong like how do we determine what's better than the other ones create 60 a 64 character long string with a letter for each piece take the fen and then compute the hamming distance a very simple metric so you're thinking in terms of strings yeah so i was thinking precision yeah so grump said precision and recount recall on pieces um i was thinking like precision recall on on squares but maybe adding some weight to squares that contain pieces in the ground truth i don't know i'm spitballing here but this is what i'm thinking so we now have two different types of boards we have this board which is the predicted we have the ground truth board which is what we know the setup is and then we need to actually get the pieces in each position so peace map what is that all right so this is our piece map in the ground truth board we have each a dictionary with each of the 64 squares and what piece is on each of them does it have the color of the piece oh that's correct that's right so lowercase uh uppercase r for this rook is on um position two hey welcome to the family ryan hope glad to have you here we're just trying to do a computer vision project to detect the positions of pieces and we're trying to make a metric so i think this is the way to do it let's just do it really crude hey ryan how'd you find the channel and i like your username python we're coding in python so um so this is gonna be very crude metric so for uh square in range 64. does this is this start at zero it's saying there's a rook in position two so i'm assuming that bottom left a1 square is zero and it goes this is gonna go up to uh 63 not 64. so that sort of makes sense because we have a a black king on g8 which would be position 64. is everyone following me here browse programming found yours about chess nice i did an opencv chess ai project before so i visited oh ryan python what link me to your um link me to your page or what your whatever code you wrote for it i'd be interested to see think positives i.e piece on a square which doesn't have a piece in the ground truth should be punished more than a missing piece maybe maybe we'll keep it vanilla here at the first and we'll just do true positives true negatives right so let's create the ground truth piece map and pred piece map and that's going to be the board piece map um i guess we could also just do peace at peace type at so if we do one let's test this out so this gives us a number a number representation doesn't it do camera stuff though just template matching on lead chess what do you mean this page on video demonstrates it oh nice let's see what you did chimp test spot what is this a bot that plays against opponents in on lead chess it uses opencv from template matches at chess library to figure out the best move in pi auto gui to play the move oh that's cool so it's like architect you don't spam the chat too much i love chat being involved here uh because if you know the previous position [Music] but this is so this is ryan python who's in chat right now his project and so if i'm understanding this correctly your chest bot um is basically automated like a cheater for for leeches like you can cheat on leaches by using this because it'll detect the board it'll detect the position it'll know the best move and they'll actually make the move that's pretty cool that's pretty cool i think i've had i've had that idea as a project before but never executed on it most of my projects i haven't actually fully executed on but all right what should we do gt board we can do maybe it's cleaner to do it this way so remember gt board is uh ground truth and then board is prediction so peace what's the difference between peace type at gets the piece type for a given square so that would be like [Music] four and then get piece at oh actually gives us a rook uh like it prints it but it's that uppercase r if we wanted to like both work but let's just do psat because if we want to do any sort of special scoring for certain pieces uh we'll have to do it that way so i'm going to cut this so this is the ground truth and this is the pred piece and we're going to keep track of a few things so we're going to keep track of true positives false positives and false negatives [Music] uh let's say true negatives ah it's a little weird here because what's like a [Music] really cool polish thank you for sharing yeah exactly ryan ryan has a great project there um so how are we going to grade this all right maybe we might need to make this simpler can we just do gt board get uh oh wait peace map and then we can make this into a data frame i love my pandas import pandas as pd we should put that at the top pd data frame on this um if using all scalars you must pass an index i think this is like this and transpose this this is one way to do it so the index is the position and this is gonna be we're gonna make it we could also do this data frame from dictionary it's still going to make us do the oh i think i need to change the orientation orient is not from columns other otherwise pass index oh so this piece map has piece and the color so this color is actually like white or black i think i need a map of this to the the square names could you just convert the boards to 3d arrays and do cat cross entropy is there like an array i know if i print it like the string representation this was it like this this looks like this and then if i import numpy as np can i make this into an array not really this is like a hacky way of doing it and split on the space let's strip out the slash ends i don't know what i'm doing are you using the python chess library i am you could one hot the string please wait a moment you will get it okay [Music] um you mean so do you think i should keep on doing it this way because i i still feel like it's it works well to do it this way i just don't like that the piece map because it's the color and the piece type so let's just okay let's continue down this road just to make sure that um i know i know how to do it this so so let's do ground truth data frame is this and then the board piece map is going to be pred data frame and then we're gonna merge these together with the suffixes of um ground truth and prediction uh left left index equals true right index equals true all right so now we have the predicted piece and we have the ground truth piece for each of these and we can see which ones are correct or not um we kind of need to join up the color with this right i feel like this is i could do all that regex stuff uh wolf i agree but doesn't this work now one other thing i need to do is make a data frame with the index is a range from of 64. right bear with me i know this is all kind of messy so this is like our skeleton and let's actually load extension lab black oh i need to do it like this it says black is an installed pip install black let's make our code a little bit cleaner make it look a little cleaner and i'm also going to do these imports pandas as pd import numpy snp i'm sure we'll need those at some point let's just restart this from scratch make sure we're getting everything right probably should have imported that okay there we go this is going to clean up our stuff a little bit because don't need this anymore this will take our skeleton and we're going to merge it on this left index is true right index is true and then how is gonna be outer here i guess we don't really need this uh oh this this merge also needs to be an outer merge right because if we have a piece that does not exist on one or the other that's that's no bueno uh let's fill in a with uh just blank and do you guys see what i'm doing you could do data frame coordinates in this supply landa np unravel index eight by eight uh so that's to reshape it in eight by eight can we join them on making false ones negative integers please wait for a moment you will get it okay so let's stop and wait for a moment what's our approach gonna be here data frame approach which i always prefer or doing uh doing like a matrix difference let's do this what i think here just let's see if the is is zero a piece number maybe i should fill this n a with zeros because i really want to just join this with the color i'm doing a matrix difference approach now if that helps you decide yeah let's see this df chord that coordinates df applied let's see this oh this is the way of um what does this actually do it's not letting me apply this unravel index on an integer index akash has a data frame question i'll try to answer it look how much i'm failing here um i know this is messy but this is what i'm gonna do yeah ask your question hey by the way if you're watching this and you don't know about my youtube channel exclamation point youtube will and twitch will bring you to it it's right here i released a new video last week i've been doing i'm streaming this live um talking about newbie mistakes that pandas users do so check that out if you haven't already um this should first be an int [Music] that's going to be our ground truth and then i can do the same thing with our prediction because the because the actual prediction is a combination of the color of the piece and the um and the piece itself hey welcome to the family someone uh let's see who joined who joined welcome to the family uh api timing welcome hope you're doing well tonight um hey chris we got chris uh chatting in the youtube channel your new pandas mistakes video helped me just started in data analytics this year after a few years of engineering and your channel really useful awesome thank you chris tell all your friends post it on social media i'm learning like all of you as you can tell but i'm just trying to share what i've learned um and honestly it's kind of selfish because i just want to learn how to be a better communicator and i appreciate all the feedback let's call this the prediction board so now this is a function that's really messy but it works like this is how unfortunately how my brain works sometimes is we give it the ground truth board we'll give it our prediction board and now we have a data frame that has like the ground truth and a prediction like we could just do this to we can clean up later so that actually shows the um right um okay so we got some nice stuff do i okay i have a data frame with the latitude and longitude columns and i have a list of specific latin two logic logicones i want to filter out the list is really long i want to type them manually the latin log2 are multi-index format to how do i filter out the most efficient way hmm as with most things with pandas there's like infinite number of ways you could do things i do things that maybe aren't the best so you could like reset the index so you have a latitude and longitude column and then query based on that like then have like you could concatenate them together into a string and then have a list of strings that you filter out on the problem is that you have it in your index and filtering on the index can be a little bit tricky especially with multi-indexes so is there is there any reason why not to do reset index make them as columns and then do your filtering there and then what you have a list of specific latitude longitude points you want to ignore or is it just is it something like within a certain distance because if you're doing that you might want to look up like geopandas actually does this stuff really well i've used before so geopandas at like is meant for working with latitude and longitude data there's also like uh spacey or something else not spacey another uh package that uses uh distances that you can calculate and stuff like that it meant for um locational data so i'm sorry that's my suggestion i don't know if it's great it's uh hey welcome to the fam mike thank you for joining us spacey is nlp though yeah not spacey i'm i'm being spacey here by saying spacey um there's something there's another library that can like in pandas-ish way calculate distance from things uh pie geo pia pigeosity oh that might be it no wait i didn't i pasted the wrong thing pie geodesy oh is this a good one yeah send this package to all your flat earther friends people so that they know that like calculating distances and stuff it it doesn't work if you don't account for the curvature of work the earth all right so we created this joint data frame and then if our predictions equals our the ground truth oops i didn't mean to do it that way right so now we have a bunch of trues and falses um see what is a false positive in this is a false positive do we want to penalize saying a piece exists somewhere that's incorrect probably probably want to consider that a false positive um so this would be like a true positive it predicted it and it predicted it correctly that's when the prediction equals the ground truth if the ground truth equals this zero zero zero and the prediction does not equal the zero zero tell me if i'm doing this in like a inefficient way which i'm sure it's not then this is going to be i guess we would call a false positive oh no a false negative because this means no this means the ground truth is nothing and the prediction is not nothing false negative that's correct i think we'll go back to that then we're going to need a false positive that's when the prediction does not equal this and the ground truth does not equal the prediction hey i suggested making the values negative if the color is black makes false negan false much easier and then i would just make it zero if it's okay so that's a good idea that's a good idea um so this basically what you're saying is basically take the ground truth oh sorry let's take this and return the full data frame that's a really good idea let's let's do your idea so we have a color ground truth if that is true what's it going to do if it's n a if it's true then let's locate wherever it's true and then we'll make the piece type ground truth how would we invert the multiplied by negative one does that work maybe you don't need to have both naming in the cases one two three if n a do nothing since it's zero yeah so now we have what we are having is peace type ground truth yeah this is a better way to do it let's see how this works piece type ground truth is going to be it's going to be negative if the color is true i don't know if we want to make it if if the color is false then it'll be negative or i don't it doesn't really matter piece type um prediction is going to be the same and then we're going to return then we wouldn't need any of this crap right and we can return this data frame which is peace type ground truth and peace type prediction how's that look to you guys right um let's make them all as integers so now we have negative if they're the right color i feel like the names true negative true positive or not making a lot of sense and just causing confusion um true yeah i guess we don't need to actually calculate f1 score anything we could just do like a okay all right you guys are keeping me in line so we're gonna do we're just gonna import from sklearn metrics let's look up sk learn metrics um the thing is i'd like to have like a confusion matrix multi-label ranking metrics no classification met metrics that's if we have like probabilities but what if we try this classification report uh how did i get way up here so classification report um peace type ground truth is going to be our truth and then peace type predictions going to be here and it hates me this is gonna give us everything so this is like the precision recall of each of the piece types so what's negative one is that the so this is cool so our average f1 score is 0.84 i kind of feel like the names true negative black pawn is negative one hey on bandit welcome to the family into the channel glad you're part of it let me know and chat how you found the channel that'd be great this is cool though we have like we have some idea of what we would want now i kind of feel like i want to wait i want to wait differently um so if i did if i import here f1 score if we just did straight up f1 score on this is it going to give us 0.84 it does not like this okay so we have to do average we have to average because it's not a binary classification so we need to decide between micro and macro f1 macro i think we want micro so each square is kind of given its own weight but what we do want to add here isn't a weight sample weight all right so if our piece type ground truth equals zero uh doesn't equal zero this is gonna be as type int so it's gonna be a zero or a one um a zero if it's i guess i could also clip this no i can't because it's i could absolutely clip it absolute of peace type ground truth and clip between zero and one same thing same result hey just stumble upon you on twitch and love how you use python so i stayed nice thanks for joining the family that's what we call it when you uh when you follow here you're joining the family also check out exclamation point youtube and and give me a sub on there i'm trying to hit 10 000 subs can we do it can we get to 10 000 subs on youtube not tonight maybe eventually um all right so i think we have some sort of a metric now the thing that i'm talking through here is is adding this weight so i if it's clipped like this we can add one and then multiply by two or something oh then we need to do it this way this is just like a clue like sample weight we're just trying to make it so that the empty sk squares are weighed less than less than an uh a square that has a piece does that make sense ooh i do you can farm the hans neiman cheating controversy i i agree if i was good enough to do that hey robert e by the way welcome back it's been a while since i've seen you i hope you're doing well um that would be fun but i feel like that ship has sailed like the content came comes out so fast that you have to be on top of these things um yeah so delete all this crude metric stuff but now what we have is an f1 score which i think the micro um is the same thing as an accuracy score so let's do an accuracy score [Music] for this and our classification report is pretty nifty um [Music] if we just did an accuracy score without any sample weight if i need to yeah it's the same thing and i learned this in our last competition that i launched because i forgot that with micro accuracy f1 score um it's basically accuracy score unless you have the potential for false [Music] negatives like if everything's gonna be labeled then uh we're basically doing an accuracy score with uh the sample weight of our weight here which is just kind of like a dumb way of doing it so let's say this is our scoring so let's say let's get our our score from by doing this providing a ground truth board in our prediction and we're going to return our score like that okay cool cool cool cool uh move these up here we don't really need the f1 score in here anymore then what are we doing deleting all this because of the great suggestion someone had to make the um black pieces considered negative or maybe it's the white pieces one will one of the colors is opposite and then we'll any in this function get the create the joint predictions and get our score so if we do get score on our ground truth board in our board we get this result cool uh confusion matrix grumpy you suggest we could check that out oh you guys are saying some other stuff have done some mean square error loss version if interested fairly basic wait so how would mean square error work on a classification so you one hot the board and then you do a mean squared error loss this is cool one hot board does this 13 by eight by eight they're 13 pieces okay but but what about the colors is it still 13 am i crazy um so anyways you're doing this you're making your board how did you pull out the i love by the way that you could just hand me this collab stuff uh let's add it here the thing about oh not loose the thing about scoring like this is like there's no limit to how many metrics we can use why not try some more i like this code though so you did the pieces like and you already have these pieces written out that's nice and of course i'm losing everything this is what collaboration looks like people collab so in theory i should be able to run this and then run this on my board ground truth board my board oh i don't want to do this what did you do what did you do you one hot encoded them first yeah that's right um and then i'll return these two encoded 1.38 is that sound about right uh six white pieces six black pieces one empty you're right i'm dumb i have a question hey darronos welcome to the family so glad to have you here we're making chessboard computer vision stuff hello quasa what's up uh hi i'm kyle said i have a question what are you using the metrics for since our case even 99 is never sufficient well because i mean we want so let's say that let's say that we have a really good model that is a hundred percent most of the time when it does fail we want to at least know how much it failed by how are rob i like the way that you format formulated that question how are rob i'm feeling pretty chill tonight we because we got i started kind of late and now i'm going kind of late um but it's a lot of work we made a lot of progress and i basically found a library that does what we're trying to do but isn't perfect and leaves room for improvement but still is pretty awesome all right so and and like i said the biggest thing that we have to do is create this metric so that we can actually score how well it does um i like this we can do both metrics i like this git score uh so if we can see if things are perfect or not so i wish there was a way for me to like set up a board i guess i couldn't go to chess.com i can go to analysis board [Music] analysis and i can set up a position here and then i can download the fen it's kind of annoying to do all these steps set up position and then i can um set up my board and then check it that way uh does the library give you the same result if you run it twice so that's what we're getting to so this might be a good thing the end nonsense is it getting late for me is to let's set up the board in a few different states and then see how well it predicts so uh right now we're doing it this way i wish i could print this board a little bit easier [Music] i'm gonna release this ah it doesn't want to pull from that is that because i have the camera open nope let's restart this just to be sure because now we can print our image or print our board [Music] and print our score and just run it a few times because it's kind of hard to check each thing right every time we do it and now that we have a scoring metric it's going to be a lot easier so this is our ground truth board um make this bigger for you all print ground truth board i need to release this and this hates me it doesn't like my webcam right now the secondary webcam i don't know if it's because the obs is up open all right so that's the board we should we should print some like uh prediction and then print a space maybe print ground truth perfect match for my msc version would be output of zero worst case is 4.92 that's so you could rescale it okay that's probably a good idea to rescale why is it hating on my convert color all right there we go um these are the two predictions and then we're going to get our metric so let's put our metric up higher this is our scoring metric score code and this is the get the score code we'll put up here we will also put in our friend here the architects score we will print so get score of our gt board and our motherboard and we'll also get our mse let me see loss yeah this is my msc board then i need to print these accuracy score and then we'll give this like four decimal places about right maybe we'd print this before we actually plot the image and do all this other stuff like put this up higher and then we will do our msc score like this should i do the dividing here like to divide by f 9840984 mse score is 1.5 0.14 so that's like yeah can i zoom in a bit more absolutely um so this is no longer that pretty representation of the board but this is telling us our score is 0.79 this granted this is weighted did i weight our accuracy score get score so we can make the weight none to start with and then we could actually score it at different weighting scales but it's a 0.85 and then a 0.147 so it's like those are very comparable right are we just computing the same thing no because is it the same thing is this zoomed in by the way enough for you guys all right so now we have our board set up right and i can like what should i do to try to improve this can i can actually move the pieces i know you guys can't see it here but i'm going to try to like move this up a little bit more maybe get some better lighting maybe that's up too far and i'm going to rerun it and we're going to see if eight five nine hey it's gotten a little bit better eight seven see now we can quickly see if it's better oh but it's actually cut off part of the board so how did it do do better bring the lamp i love lamp anyone get that reference uh .5 why is it so bad oh because i cut off the whole bottom so the prediction has like none of this bottom so i'm zoomed down a little bit more see so like we can quickly now see how well it's doing that's why we're making this metric and we have like a we have a pretty uh standard way of of evaluating it 0.82 what's it getting wrong here well pieces are not necessarily correctly on their squares [Music] another thing would be like a heat map to show us where it went wrong can't what's the perfect way i can get this is it getting this king it's still thinking this king is a queen and this rook is a knight 0.8954 let me try to let's take this image and crop it a little bit first let's get this situated so everything's in view like this rerun this there it's mostly in the view and we'll do like 100 to 400. i guess we need more at the top like 50 and then we can do 150 to i guess that's gonna be enough not sure if you've tried other solution than chess cog i think the ultimate goal is to like do it our own way oh this is cool neural trust chessboard most of these angles are pretty high but that's really impressive we can now we can now score them against each other these animations are cool but this is five years old i can't imagine it being the best right the other one released a paper talking about if this is five years old i'm doubting it's gonna be better than the other one but next stream maybe oh yeah baby all right so let's crop this image and see if it does any better eight four three eight it's still still doing poorly on these last pieces um [Music] can i put something in the ground i don't know i don't have any paper or anything uh let's do like let's end on this note let's try to just set up the board in the standard format that you would set up a board and see how well it does and then uh end the stream sound good it was kind of a chill stream tonight but i hope you guys had fun i had a lot of fun i learned a lot i really think that we can we can find improvements on this i think if this was a kaggle competition and maybe there was a gpu involved that we would see some interesting solutions it might be better the older one was trained on images of actual boards with varying piece shapes etc which is closer to your input that's true that's true yeah this one was trained on um on synthetic images next stream we will test that out i don't i don't trust myself to uh to test it right now because then i could see myself being up for another four hours [Music] right the board is set up into opening position i haven't changed the fen of this so this scoring metric is going to be screwed up 0.46 see i think it should be a lot worse than 0.46 for what it is um [Music] started watching my opinions videos a few days because though because i didn't know how to use a group buy and have been loving your stuff decided to check out your stream and it's been great thanks so much the architect i can't believe that you needed my videos to do better because um you've been helping me out all night so i love it all right so let's get the fen position of like a base uh standard board that's gonna be uh reset uh then we're gonna download share this fen position so this is our old that's our [Music] now our ground truth is this and it's oh wait i still have this queen down here i think maybe adding object detection to this might help just doing object detection could be cool so it's screwing up it like can't handle the fact that these pawns have other pieces obscuring their view i'm surprised that it does perfect wait wait yeah i'm surprised that it oh it thinks everything at the top is a night um so let's also just do this oh that's the that's the real problem so we have big problems here we have big problems here it really likes them nights um i feel like it's just the background being dark let's try one more thing um okay am i an idiot here is this a stupid idea but i'm gonna take i'm gonna take a paper towel and put it underneath the back of this board that's not gonna work that's not gonna grow contrast is good yeah but like ideally we would want this to work without without this added contrast look at my paper towel it's not helping the lamp brightness might actually be hurting it here's a second lamp i'm going to turn on um and then i need something to need something to hold this up my roller [Music] all right this is silly this is just absolutely silly but we're gonna see [Music] if that improves things now we have like a white background behind the black pieces and it still pons knights rooks thinks the king is a queen let's let's actually print this out so it doesn't think they're all knights but we should expect better right better than this uh still missing some of the pawns for the white pieces i think you're right whoever suggested that other architecture is probably going to beat this so at least in this example because we don't have simulated things all right cool so bring this to here thanks everyone who watched on youtube today and uh on twitch it was a lot of fun we tried a bunch of stuff and it actually turned out better than i thought the main thing was we figured out a metric that we can use to classify how well our model is doing now i could have used a a better quality camera too and we could have just pulled in images that we had already pulled in from uh game footage because we've written all the code to do that from that just scrapes youtube videos and see how well it scores on that um but we have a baseline here we have a baseline here which is a really nice art um uh paper that was released on on this topic thanks everyone for hanging out also maybe they train on high res images as they use blender so it might be confused by low res true yeah we could we could there's a couple different ways we could take this we could use blender to create more images add in a little bit more augmentation um the hard part about making our own real images is like take a lot of time to set it up but i've thought of like we could try to do something smart with like put each um piece in each position with like a green screen in the background and then do some sort of like auto creation um there's a lot we could do or we could just make a competition see who does the best in in finding the results i'm voting for that one and maybe have some prizes on the line okay y'all we are going to end the stream now um maybe do a raid and then um there's a lot of coders on tonight uh i know we laid we we raided uh nick last time and he's not coding right now chrissy codes i feel like we got to switch it up a little bit cyforce one i also like raiding but cyforce one is um wait he's hosting me right now what's going on uh it looked like he was doing react stuff now let's raid chrissy oh she just got raided man i overthink this [Music] [Music] oh cyforce you just raided how did i not get the alert i was about to re i was thinking about raiding you until i saw you were doing um react but i appreciate the raid guys we were we were working on uh chess board vision computer vision stuff so if you look right here looking and currently looking at who to raid but uh we we're running this on a webcam that i've set up here and trying to detect the position of this chessboard and the the pre-built model is not doing too great it's got a lot of queens down here that it's predicting uh it's not perfect but it's uh it's a been interesting to try try and figure out yeah no problem but thank you for the raid i appreciate it i hope you guys check me out oh by the way before i leave exclamation point youtube brings you my youtube channel uh trying to get to 10 000 subs on there getting close and uh discord is our discord channel if you want to say what's up on there and then uh exclamation point kaggle is my kaggle account we may be releasing a new competition soon so be on the lookout we have nice prizes that we give away like gpus so uh check that out and it it should be fun so uh back to this back to the rating i just have to make a decision i like need a i need like a spinner wheel that helps me to figure out who to raid because i'm so indecisive um [Music] but let's go ahead and raid time enjoyed so thanks i've enjoyed my time with you all i hope you have a great night and uh yeah be kind to each other love you all let's start this raid moomi thanks for hanging out that i know you're new um hello