CoPilot Review: My Thoughts After 6 Months

Key Takeaways

so I've been using copilot now for six months and I have ran into some pretty world-class bugs that took quite a bit of time for me to figure out and I figured out ways to use it in the most efficient way possible and I've really enjoyed it so let me take those six months of learning and just give you a few points on how to use copilot and use it effectively and more so let me kind of address whether or not you should use it because I do think that not everybody should use it if you're unfamiliar with copilot it simply is some sort of AI generation of code so if I go function subscribe to the primogen and I want to return a Boolean it's gonna go through and tell you the truth so go and do it the robot says so so first of all rule number one don't do comment driven development it almost always is a terrible way to use copilot like when I see someone going uh write the quick sort uh quicksort function for me right or however you want to say it it it's gonna go over there and it's gonna do a good job you're gonna have to give it a bit of prompting well guess what in the end it's going to produce quicksort and guess what that is effectively correct I can tell right away but I can also tell a couple other things when you let co-pilot just Jesus Take the Wheel style it's gonna do things that are both correct and incorrect now I want you to think about quicksort it's an extremely well documented well understood algorithm and if you look at this there's something very very wrong about it but it's not logically wrong it will produce the right answer what is wrong about it is that a it creates two arrays here and it creates an array here it actually programmed quick sort but defeated the reason why quick sort is so fast quick sort does not require any memory allocations that's one of the reasons why it's so amazing this requires several memory allocations in fact if I'm not mistaken this is an N log n memory calculation going on here that's not good so rule one don't do comment driven develop development alright so rule 2 will take a moment for me to show you I've been able to reproduce this bug super reliably and it's come in many different forms so I'm gonna go like this struck by the way we're using rust right now because you know rust is just the greatest language of all time there we go a video frame is a time stamp and audio frame has that struct video is going to be oh that's still typescript frames it's going to be able to kind of guess what's going on there it's going to be able to guess that all right so now I want to implement the frames per second calculation for video so what I'm going to do is I'm going to go imple video and I'm gonna go Pub function FPS take a reference to self return out a float64 and I'm gonna go return um match uh self and it's going to go through and it's going to be able to get out some good information right here and boom we have an FPS calculation there's actually a couple bugs here that have happened I don't know if you can spot them right away but let me tell you two things that have happened first off what is the unit of time for our time stamp we never specified copilot doesn't know it actually is considering it in seconds watch this if I jump up here and I rename this to type milliseconds jump back down here erase this code and re-let co-pilot fill it in it still didn't even get it right in and out of the syntax error what happened here what's wrong with this well to be able to get frames per second you need to multiply by a thousand to convert your milliseconds into a seconds we're not doing that here that means if your frame links was 30 and your duration's a thousand Thirty over a thousand that's not that should be thirty it's not 30 but second there's a way more Sinister bug going on here does anyone see it really look for it what could it be well let me tell you what is wrong with this statement the statement says hey I'm going to grab the first video frame and I'm going to grab the last video frame what if the first video frame is the last video frame what is the duration the duration is zero what is going to happen when we divide by zero well I can tell you this much not a good thing it's not good to divide by zero in non-javascript languages at least in JavaScript you get everybody's favorite number which is not a number so this leads me to my second point which is I almost exclusively never let co-pilot write my functions I like to start writing them and I will let copilot do a bit of completion for me so if I were to redo this I would do something like this return match self let co-pilot do that because I'm fine with what I see here and I don't even look at what copilot's gonna do I just delete everything inside of there so I just give the old the DI squirrely brace just get that out of there because that is crazy talk after that I'm gonna add the things that I want self frames length put that in there and now copilot gives me a new suggestion guess what do you still see it the bug is still sitting in there so what I'll do is I'll accept all this remove the syntax error remove all of this and then add a little beautiful piece of logic if n is greater than one look at that now this is fantastic this will actually give me the correct answer and then of course for everything else just return zero so now all I have left to do is really look at this since it's a very simple piece of code I should be able to just look at it and see it and know what went wrong but again the more code you let co-pilot write by itself the more chances you're gonna have an error now luckily it notice it was able to catch on to this whole notion right here notice that it went from milliseconds to seconds and it did a proper unit conversion and then now is doing the division but I kind of want to rewind a little bit here remember up here when I was doing the audio frame and how the audio frame just like bam came into existence well once you have a template of logic and you want to really reproduce it logical boilerplate is incredible so if I were to take this and go imple audio and I did Pub function it would pretty much be able to guess I want the exact same thing and it really does a great job of taking what I had up here and spiritually if you will moving it into here and in this case exactly what I had up there I find this to be one of the best ways and the third point which is Define what you want and then use copilot to do the boilerplate here I can show you another example of that watch so if I go like type a media here we're back in typescript to make it easy on everybody I just want to show you that good rust I gotta work in Rust somehow people so let's say we have an ID title description URL type created that updated that use right why not now let's not do user okay we've done enough I can now hint to co-pilot how to create this boilerplate for me so I can go like this create media by using the name of the type in the function name it's going to kind of know what to do but even more so if I go like this ID number and then tell it what I am returning it almost exclusively always gets this next thing correct I don't know why it didn't just get it correct Come On Media show me what to do here return yes oh come on go pilot why you gotta be like that and let's go yes ID absolutely yes title title description description url url type created at updated at and user ID it's really great at creating some of this boilerplate and you can imagine if you have to do some sort of like object data transformed from say a database Source into a custom object if you have these two types defined co-pilots really great at mapping these things together other and this is just one of those things I find so amazing so Point number four if you want to use copilot well make sure you have your types as explicit as possible because the more explicit your types are the more co-pilot can follow what's going on up in here but now the real question is should you use copilot and I think this is a great question to ask because it's not a yes or no I think it largely depends on where you're at in your career honestly if you're a junior I would just so recommend you don't use copilot right now if you can't easily spot bugs if you haven't been in the workforce for a long time it's gonna be so hard for you to capture all this and if it's not captured in a good code review you might find yourself just letting bug in after bugging and it's going to cause a lot of emotional grief for you and not only that but you're gonna find yourself watching and code reviewing so much more frequently inside of your IDE that it's just gonna make the experience of creating software a lot more awful whereas if you are at the tippity top of your game you're smooth as coconut oil you're going to get in there and you're going to start writing and it's just going to help you smash out any of that boilerplate so fast and there's sometimes it's just wonderful I just love that part of copilot so am I going to continue to use copilot I'll probably give it a little bit longer of a chance and I really am just going to keep on reducing any logic that it produces versus copying ideas or filling in types for me I just find that the sweet spots the boilerplate it's not the logic the logic is almost either flawed wrong or just really really really inefficient I hope you press the like button because this took me six months of just crushing bugs from copilot to be able to make this video for you the name is the primogen