Revealing My New AI-Powered Data Science Workflow

Key Takeaways

Okay, so my data science workflow has completely changed over the past couple of weeks with the introduction of tools like chat GPT and also GPT-4 model and GitHub co-pilot. It's just not the same anymore and working with these new AI-powered tools is really a skill on its own. So I've been really practicing with it, putting it to the test, being trying to be creative with these tools when it comes to my data science projects. And in this video I'm going to reveal my new workflow. Basically everything that I've learned, how I interact with these tools to basically complete my data science projects as fast and efficient as possible. So this really is a complete game-changer and these tools aren't going anywhere. They will only get better over time. So really if you want to stay relevant as a data scientist or engineer or analyst, you really have to learn how to use these tools effectively. And this video will help you with just that. So let's get into it. Okay, so in order to keep this as realistic as possible, I have looked up a Kaggle competition, basically an old Kaggle competition where we can download a data set from. So it's about a multi-stage continuous flow manufacturing process and here basically is described what the primary goal is and it is to predict some measurements giving the data. Now this project is data set is new to me. I've only downloaded data set, looked into it and see if it's relevant for what I want to do, but other than that I'm completely new to this data science project, this data set as well. So what I want to do with this is go through it brand new like I would start up a new project and then go step by step and show you my thought process and how I interact with chat GPT and also GitHub co-pilot to basically deliver on the primary goal, predict measurements of the output from the first stage. And we will get into this in a bit, but that is basically the setup. So, I will share this document with you in a description. Here you can also look up the Kaggle competition. You can also download the data set if you want to follow along. And throughout this video, I'll be using my data science project template, which you can download in the zip file over here. If you want to learn more about that, you can check out this video. So, the best way to organize your video, and we will be using VS Code for data science. And if you want to learn how to set that up, you can check out this video. And then for our AI tools, we'll use GitHub Copilot and GitHub Copilot Labs. And if you don't have that set up, you can check out this video. So, how to use GitHub Copilot for data science. In that video, I will cover everything. And then finally, we'll be using Chat GPT. So, I am on the paid subscription, so I will be using GPT-4, their newest latest model, which is a lot more powerful for coding than GPT-3.5. But if you're on the free version, you can still follow along just fine, but I'll be using GPT-4. All right. So, I am now in a brand new VS Code workspace. You can see we're using the data science project template, and I've The only thing I've done basically is I've downloaded the data, so the CSV file, and along with that comes a notes on dataset.txt. So, here we can see more info about the dataset. So, that will be useful in a bit. But before we start, as with any data science project, we first have to create an environment that we want to work in. And that is the first step where I'm going to ask Chat GPT for help. Because I don't know about you, but I always tend to forget the specific syntax and basically the commands, the prompts for Conda to create and work with environments. So, I always have to look those up. So, the goal is to create a basic data science Python environment from a YAML file. So, that is one what we want to do over here. So, I come back to chat GPT over here and I'm basically going to create the first prompt. All right, so here's the prompt. Create a conda YAML file called environment.yaml with the most common packages for data science. I specify that I want to use Python 3.10 and I want to call the environment manufacturing process. All right, here we go. So, now it's creating the YAML file over here. And GPT-4 is a little bit slower than GPT-3.5. So, while this is loading over here, I'm just going to come back to VS Code and open up a new terminal over here and do a quick touch environment.yaml. And then I can come back to chat GPT and we can see the result over here. So, we have a nice setup for a YAML file and we also have the specifics of how to install this. And like this is the command that I also always forget. Like where where does the F go? Does it include env? So, this is just really useful. So, now I'm going to just I can just copy this, come back over here and I can already tell that there is quite a lot in here. Probably a lot of stuff that we don't need for now. So, let me just get rid of most of that. So, here you can also see I'm not just copying everything blindly. I really look at my own specific situation right now and see I want to first start off with the basics and we can always add more libraries later. So, also come back over here to basically pre copy the command to create this environment. So, first let's save this file over here and now since we're here in the terminal already, I can call conda env create and make sure that your terminal is in the same directory as your environment.yaml file and now we can run this command. And now it should load everything up and ask us if we want to proceed. All right, so the environment is now doing the installation. I got rid of some more dependencies because I'm not sure, but the loading took very long, so it was stuck on solving requirements. So I got rid of some stuff as well. So now you can see it's installing correctly and it should be done any second now. All right, and it's done. So what we can now do, we can come to the source folder over here and I have already a setup in here which is called make dataset.py. And now from here on, since we're working in a Python file, we can now select our brand new environment. So here we can see Python 3.10 manufacturing process. So we can select the new Python environment and then make sure that we can run everything. So let's see if we can fire up an interactive Python session over here. Let's get rid of the terminal and then see if we can load, in this case, NumPy, Pandas, and Matplotlib. All right, and the imports went fine. So that means we now have a brand new environment that we can work in. So let's load up the data frame and see if that works as well. All good. So first step out of the way. And I'm also going to quickly install Black just like VS Code is asking me for because I use Black as an code formatter. So let's just add that as well and make sure to update it here for reuse later. So if you don't know what Black is, it basically allows you to automatically format your code following the PEP 8 style guide. And you can set this up to automatically run this whenever you save a file within VS Code and this is just really helpful to keep your code nice and organized. All right, so now that we are basically up and running, let's come back to the original problem statement and figure out what it is that we actually have to do. So again, this is also new for me. I've looked at it briefly, so I know that there are two goals. So, we have a primary and a secondary. And for now, I just want to focus on the primary goal, and that is predicting measurements of output from the first stage. So, we are dealing with a manufacturing process, and it contains various, let me see, what was it? Machines. So, for the first stage, which we will be focusing on, there is a machine one, two, and three, and they operate in parallel. And they feed their outputs into a step that combines flows. So, this is probably some kind of chemical process that is happening, and eventually all the flows from one, two, and three come together. And then, once they are mixed, I guess, they measure in 15 locations, and these measurements are the primary measurements to predict. So, you can basically see we have three machines. They do all kinds of stuff, and then the flows come together, and then we have 15 measurements, and that is what we want to predict, basically. So, this company is basically trying to figure out how they can create some kind of a simulation or process optimization model to figure out what should happen in machines one, two, and three in order to optimize for the output that is coming out of these machines. So, that is all that I know for now. So, let's have a look at the data and try to figure out, basically, what it is that these machines are, what they measure, and then also the outputs, what these outputs are. So, let's come back to our data frame, and let's first run a data frame.info, and see what's going on. And here you can see, since there are a lot of columns, 116 in total, it doesn't show us the columns. But, what we can do is we can set the verbose equals to true. And here you can already see GitHub Copilot doing its thing. So, GitHub Copilot will be more subtle within this project and I will illustrate when and how I use it, but the main thing where where you see the big things happen, that will be in ChatGPT. So, we can run df.info and then verbose is true and that will show us basically all the columns that are in there. So, the 116 in total. So, we have a timestamp, so this is time series data. And let me also quickly grab the notes over here. So, here we can also see this is a description provided by the company as well. So, sample rate is 1 Hz, which means there is a measurement every second. And then we have the machines operate in parallel, that's what we've seen. Measurements could be noisy. We have set points and actual values and we have some ambient conditions. So, this would be probably be the room that these machines are in. So, we have temperature, humidity, makes sense. These are actual values. Okay, so here we have machine 1, 2, and 3. And we have raw material temperatures, all kinds of measurements and then let me see we also have the outputs. Okay, so here we have the 15 outputs that we want to predict and here you can also see there is a set point and there is an actual value. I guess we can first trim this dataset down to only include the columns that are relevant for the primary goal. Meaning predict these these values of these 15 locations and then not use the set point, but use the the actual values. So, let's come back to our dataset and then basically say selected columns equals and we are going to create a list over here and let me see, can I just I am going to basically make that selection based on the columns from the data frame, and then I'm going to say I want them up until what is it? We want 71. Does that make sense? I think that is correct. So, those are basically the columns that we want in there. And then we probably want to trim that further. So, let's first put this in here, so we don't even have to create it manually. We can do it like this. And then we can say DF equals DF selected columns. So, now we should have a subset over here resulting in 71 columns in total. And then we also have the And now if we look at the We can basically run the same command again, and now with less columns in total. We also have a lot of set points in there that I'm not particularly interested in. So, here we can see or here I can show you probably how we can use GitHub Copilot. So, I can create a comment over here and basically say, "Get rid of columns that contain And it is set point. So, get rid of columns that contain set point in their name." All right. So, now I can say basically DF is And now probably GitHub Copilot will start to do something. It will say, "Give me the location of the data frame where DF column string contains set point." And then it negates that. So, let's see if that works. So, 57 columns. And let's save that, run the info again, and now we only have the actual values in here. So, it worked. So, here you can see basically I can work with GitHub Copilot by just writing a comment first, and then GitHub Copilot basically already knows what I want to do. So, it's basically a prompt that you can also put into Chat GPT, but it's faster to do it like this. So, you really have to figure out like the nuances like when do I use GitHub Copilot and when do I use Chat GPT? All right. So, the next thing I see is that we have a timestamp in there. And that is currently stored as an object, meaning that Pandas does not know this this is an actual timestamp. So, we basically want to convert that and look, GitHub Copilot is already suggesting, "Hey, do you want to convert this?" And yes, we definitely want to do that. So, we set it like this. So, what we can do if you run this and now look at the timestamp again, Pandas now understands that we're dealing with a timestamp. And since this is a time series data set, we typically want to set the timestamp as the index because this unlocks some of the functionality from working with timestamps within Pandas. So, let's do that right now. And basically say like DF and then here it already knows, "Hey, you probably want to set it as the index." And now we have the timestamp index over here. All right. And the data set looks pretty good for now. So, you can see how within this code we have worked kind of messy in a sense that we just have dumped a bunch of code over here. One has a comment, the others don't have that. So, what I will now show you is what I would do and what I would would use Chat GPT for right now. So, basically we read in the Pandas data frame. And now I'm going to basically copy and paste all the code that we've created, come back to Chat GPT and ask it to create a nice Python function out of this code. And I just dump it in here. All right. And you can now see that it's creating this nice function for us. So, let's just come back and we can basically get rid of all of this, put in our neat little function, store this, and then it probably will also tell us how to run this. So, I can come back, copy this as well, come over here. And now, let's see if I start up completely fresh, so I start up with a new interactive session, load the data frame. So, here we have our data frame with all the 116 columns. I store the function in memory, and then I just run it. And boom, we now have our neat and tidied up data frame with documentation. So, now once you come back later to this project, or you hand it over to a colleague, you actually understand what's going on, and also basically the rationale behind the steps that we are taking over here. So, beautiful example. So, we are dealing with a data frame with all numerical values, and there don't seem to be any missing values in any of the columns, so that's all good. So, now our goal basically for this project, if I'm understanding it correctly, if we look at this from a machine learning perspective, we basically want to create a model that can take in all of let me see, these values over here, and then predict the outputs of stage one. And we have 15 values here in total. So, we could come up with an approach to create predictions for all 15 at a time, or we can create a model for each specific value, so starting off with measurement zero in this case, and then create a model for just that. And it will probably make more sense to start with that, so to start simple. So, try and come up with a model that can, given all of this prior information, make a prediction for the output of stage one for measurement zero. All right. So, whenever I do a data science project, I always follow this data science life cycle to some degree. So, I really start with the business understanding, and since we can't talk to the client in this case, we cannot know for sure why they want to predict this, but it's probably, like I've said, from a simulation or a process optimization point of view. And we are now basically in the data understanding and data preparation phase. And to better understand that data, basically, we are now going to create some plots. So, as of right now, it's pretty vague, right? We have looked at the brief description that was given on the Kaggle page. We have some info over here. We know that we have some machines, but what are we actually dealing with? So, let's briefly export this data set, and then load it into a new Python file in the visualization folder, where we can create some plots. Did you see that? This is actually pretty freaky. I said plots, and it suggested, "Do you want to create a function called plot data frame?" Is GitHub Copilot also listening? Like, what? Or is it just the logical next step? I don't know. But for now, let's store it as a pickle file. So, we're going to do two pickle, and then I'm going to store it in data, and then I'm going to to store this in the interim folder, and then just call it data processed.pickle. So, let's store it over here. Come to the interim. So, now we have a pickle file over here, and we can save this file, and come back to the visualization. So, visualize.py, and here I do the DSI imports again. So, this is a snippet that I use from VS Code. And I import the data basically again and let's see. Let's get rid of this and now we should have a file over here that we can use to create plots of the data frame. All right. So, we have a lot of columns that we have to visualize. And I have an ID here and I'm not sure if it will work out, but basically there are a lot of similar values. As in we have machine one with all the raw materials, we have machine two, we have machine three and then we have all the outputs and we also have here the first stage temperatures and I think what would be nice is to create plots where we group certain values together. So, it's a time series, so they all share the same timestamp basically. So, we can use that to combine multiple line graphs together in a plot. And now let's ask ChatGPT to come up with that because coding all of that would take quite some time figure out what should we combine and how do we specify the, for example, multiple labels that we want to include within the plot to make sure we can understand what's going on. So, let me just copy all of this. I'm going to just copy and paste this even the index over here and then come to ChatGPT. And then first paste this in and then scroll all the way to the top and make some space over here. Okay. So, this is my prompt. I want to better understand this data set. Can you provide me with code to create line plots for all of the columns and group similar data together? Make sure to clearly label everything based on the column names. And then this is something I've added and I'm not sure if this will work out, but I say don't group by the machines, but rather by the properties because I can imagine how if, for example, you group everything from machine two property one, two, and three might be on a completely different scale. Whereas if you combine for example property one from machine one with property one from machine two, that will probably be on the same scale. So, that's my rationale behind this prompt. So, let's see what it can come up with and then we'll go from there. But this is basically like a little trick that you should provide it with lots of input. And especially with GPT-4, you can I believe insert up to 25,000 tokens. So, you can actually provide it with lots of information, even documentation. And here you can see what it's coming up with. Okay, so we're going to use matplotlib and seaborn. So, let me already import seaborn. I think that was installed in the environment, right? Is that correct? Can we import seaborn? Yeah, so that was installed. So, seaborn was in there. All right. Let's come back to chat GPT and see what we are dealing with over here. So, it says group the columns based on their properties. So, we have ambient, machine, combiner, and stage. Okay. So, it seems like it is combining the machine columns, but let's just let's just see what it will do. So, we already have the data frame, so we can basically start from here. Come over here. So, I don't expect this to work straight out of the box because let me see, what is it grouping now? So, we have ambient. Okay, this so this should work, but humidity and temperature will probably be on a different scale. Then we have all the machines and then we have the combiner columns. Okay, and then we have the stage outputs. Okay, that makes sense, but now let's create a function to plot these lines. All right. Let me actually I like to make plots usually a little wider. That's just my personal preference. And then let's see what happens if we plot the ambient columns over here. All right. So, here you can see we have the humidity and the temperature. So, let's see. Temperature is the line above, the orange line, and then the humidity is over here. So, this actually works quite well since they are somewhat on the same scale even though they don't share the same units. So, this works, but now the machine properties here we are going to run into troubles. So, let's just first get a brief overview of the mess that this will create. All right. So, I interrupted it because it was taking really long, but here you can see the problem that we're running into. Lots of columns and all on different scales. So, we don't want to combine all the machines like this. So, let's come back to ChatGPT and ask it to change this. All right. So, let's update this prompt and see what it can come up with. And I always try to be pretty verbose within my prompts, and you don't really have to be correct from like a grammatical point of view, so that's really nice, but sometimes it just helps to really explain what you're trying to do. So, I say, "Don't group all the machine data, but split it up so that you group" and then I give an example, "raw material property one from machine one, two, and three, and so on." And then again, do this for all the machine data, creating groups of three columns for all the data. So, let's see what's going on over here. So, function to extract properties from machine columns. So, okay, this is pretty interesting. Okay. So, let's see what's going on over here. So, we have this new function, and it looks pretty promising in the sense that it goes through all the machine columns. So, we first split those up, and then we're going to basically split by the dot, and then we are going to create a set. So, let's actually see what's going on over here. So, we have the machine properties. Look, and now we can see that for all of the columns basically within all of the machines, they These are all the unique ones that we have. So, we have the exit zone material, blah blah blah. All This is basically everything. Okay, so that is looking really good. And now, what happens? Did Did we actually change? Okay, yeah. So, it did change up this function, so that is a new one. That is a new updated function that is used specifically to plot the machine columns. And we don't use the plot columns, but now we use the plot machine columns for this. So, let's make sure to store that, and then let's actually see what's going on over here. All right, and they are appearing. So, first up, we have a temperature, machine 1 2 and 3. And this is exactly what I had in mind, by the way. So, this is already so amazing. So, look at the title. So, this is the machine properties, and now we're looking at zone two temperature. We have a pressure. This is again Every time I use this, this is so unbelievable. Look at what we have accomplished in so little time. Like this already, and I know because I've been working a lot with process industry companies, plots like this are really valuable. And this could even be a whole project on its own, just creating plots like this from the raw data. Because this This is so nice, because now an operator can really look at how the machines are operating, and the little differences between them, and it's just really really amazing. So, here we can also see some interesting stuff going on. So, some So, really great. All right, and we're done. And we have some beautiful plots over here and really literally like I imagined. So, all the units, basically all the scales match up. So, now we can have a look at all of the data and have a brief look at what's going on. So, let me also quickly run the other two. So, that is the What do we have over here? First stage combiner operations and then also the output. So, the output will probably also cause some troubles because we are dealing with 15 values and I can imagine how they are measuring various different parameters and they are all probably on a different scale. But, let's look at what it can come up with. And again, this is really amazing how we basically created a simple prompt, it created it, and then with like two extra sentences, two extra additions to make it more specific, it worked. So, coming back to how you work with these tools as a data scientist effectively is first of all really knowing what it is that you want, understanding the underlying data, and then creating really specific prompts for that. And that is really a skill on its own and is really like the main message that I want to convey, that I want to explain, and I want to teach in this video, how you interact with that. And I thought like the only way to really teach it is to show you how I actually do it in the moment and be really creative with it. And it just takes practice. So, let's see if Okay, yeah. So, we have the combiner operation. Looks really good. That is a temperature. So, we have a temperature over here and here you can see that this is quite messy. So, here again for the outputs, we have all kinds of different data. For the sake of convenience, since we are probably going to look at the outputs individually since we potentially have to create a model for all of those. I'm just going to ask Chat GPT to split those up basically. So, create separate plots. All right. So, simply asking adjust the code to separate the plots for all these stage output columns. And then I add to that provide me with just the new code because GPT-4 also tends to be pretty verbose in the sense that it could output all of this again and then simply adjust a few lines but again just shows you all of the code. And since this can take some time, I just ask it to give me just the new code. So, here is a new function plot the individual columns. So, we can basically add this to our visualization script, save this, and then let's see what's going on over here. All right. And now as you can see for all of these stage one outputs, we will get a different plot. And here you can really start to see that we actually have a lot of noise and potential outliers within this data as well, which is something you typically see when you are working with process industry data. But hey, look at how beautiful this is. We have a nice little script over here visualize.py with four functions and then basically five lines to call those functions and create plots for the whole system basically. And again, this could be a project on its own. I've done plenty of freelance data projects where the goal was just to visualize the data properly. And now we've done it literally in a matter of minutes. All right. But now next, the goal is of course to create models prediction models that can predict these stage one outputs. And for now I already can tell that we have to do some cleaning first and we have to pick a column to start with. And it seems like most of them appear to be pretty straight, but that is mainly due to the outliers over here. So, probably once we get rid of the outliers and zoom in the range of the Y axis, then we can probably see some kind of patterns going on over here. But, basically I am scouting right now for a parameter that we are going to look at first and why not just start with the first one? So, I'm going to look at measurement zero. So, let's come over to another file where we come up with the features. So, we're going to build features.py and as far as I can tell when I look at the other parameters, so for example, the machine properties, the data seems to be all right in the sense that there are not many outliers. So, so this one's kind of messy. So, this is a feeder raw material feeder parameter. This looks quite quite messy, but it could also be just how the process is is operating. And as you can see, this looks pretty all right. So, we have some some spikes over here, but nothing really crazy going on. Again, this could also just be how the process is actually working. So, for now, let's focus on a method first to clean up this output data starting off with measurement zero. All right. So, I'm in the new file build features, reading a data frame and this is the measurement that we're looking at. And I'm doing a quick quick plot over here now just calling the dot plot to see basically, we want to get rid of these extreme values over here. So, let's ask ChatGPT to come up with a function that can help us with that. All right. And here we go. So, create a function that can clean up the data, i.e. remove extreme values, and the input should be a series, and it should return the clean series. So, this is really important. Like, figure out what do you want? What do you want to input? Do you want to clean the whole data frame at once, or do do you want to be specific and go at it column by column? So, a series, basically. So, that is what I've defined. And then, I also explained that we are dealing with time series data, where the value should not increase very drastically. So, I know from experience that when you are dealing with time series data, so let me come back. You can really tell by like visually inspecting this data that this bump here doesn't make sense. And that doesn't make sense because you can see a stable line over here, and all of a sudden, you have a big big jump. But, it is important to state that specifically that it is time series data because sometimes, if you use other more traditional ways of detecting outliers, a sudden bump within a signal could still fall within the normal ranges of the data. For example, if the data is going up and down, and you have these big increases. So, for example, the data over here would be high as well. So, it would go up and then go down, so following a sine or a cosine signal, basically. And then, it would look at the data as a whole, and basically determine, "Oh, this value of like, what is it? 2021 isn't that extreme because it's We can also see that point over here." But, it's about the sudden increase, basically. So, I'm really curious to see how ChatGPT will deal with that. So, it's creating a nice series. So, what we have? We have a window size, standard deviations. Okay, so it's doing a walking outlier detection model, basically, with a window size of 10 seconds in this case. So that that is good. So let's see what this can do for us. So we are going to define the function. And then let's quickly have a look at what this will do. Clean a time series by removing extreme values, filling in missing values using linear linear interpolation. Okay, so that is also one thing that I've added. Again, coming back to my request. I know the signal is pretty stable, so after removing an outlier you can fill up missing values by linear interpolation. Which basically means that hey, if we look at this, we just want to get rid of the values over here and just let it continue in a straight line. And now it is up to us to figure out if that is correct. So let's see now what we can do. It's actually quite funny. I didn't mention that, right? So it's already giving us the first sensor value over here. So measurement measurement zero. So sometimes this stuff is so weird. Like that is exactly what I'm doing. Okay, we can input the whole series basically, which is nothing more than just a bunch of numbers with the timestamp, and we are going to clean it by using the moving average and the moving standard deviation, and then we are going to identify the outliers basically over here using three standard deviations. And then we're going to do an interpolation. So now I guess the trick is to see whether a window size of 10 seconds is enough. So we cannot really tell from this image like how long this period lasts, but let's have a look. So we are going to define this and then run this, and then we are going to go all the clean series, and then Okay, so you can see that it hasn't been sufficient in getting rid of all of this data. So what we can then do is we can either increase or decrease the window size. So, let's for example see what if we use five for example. We still have the data in there and now let's see if we bump it up to something like 25 and maybe we have to go a lot higher over here. Okay, so by using a window size of 100 seconds, so almost 2 minutes, you can see that we got rid of the data over here. And but we still have these values over here. So, let's just like bump it up like really high. See what's going on. Okay. Okay, so you can see over here that somewhere in the beginning the value just drops to zero. And we can simply get rid of that as well by saying, "Hey, if there is a sudden zero value, which is usually from like a process sensor's perspective, if it's zero, there's just an error basically. So, we can just add that to the to the function basically. So, this is a really straightforward one. And we could have also done this probably better with GitHub Copilot. So, here you can see basically that we can set the zero values to np.nan as well. So, coming back to the function over here, we basically set the outliers. We define them over here and then we set them to none and then also we want to have the clean series. So, basically like it's stating over here. So, that is the clean series and put that in here as well and then basically let GitHub Copilot comment that replace zero values with none values. So, now if we run this, we can also have a look and then we can probably decrease that and it might even get rid of everything all at once just by look. Okay, so now we have a nice signal and I'm actually quite curious also what if we just do it like this because it seems from the data. Okay, so we You that that outlier going up. So, it seems like most of the outliers going down were actually zero values. So, we got rid of those by using this over here. And now we have to use the window size to get rid of the peak around 22. So, let's experiment with values to get rid of that. So, once again, was it at 100? I think we got rid of it. Then, yeah, let's make sure. All right, let's leave that for now. So, now this seems like a pretty clean signal that we can work with. All right, so let's store that in the data frame itself. So, we will override it. And now here we can have a look at the data again and we're now good to go. So, if we further continue with this, we now basically for the first part are going to get rid of everything other than the measurement zero for the outputs. So, we're basically going to say that, "Hey, we want the data frame." Okay, so let's get it up until column 42 because that is the measurement zero value. So, let's store that in the data frame as well. And basically, why I do this is to make sure that we only have the predictor columns that we are going to use and we have the target variable now. So, that is why I've set it up that way. All right, so now that the data is clean, we don't have any missing values, we are going to ask if ChatGPT can provide us with another function that can add additional features. So, again, I'm going to copy everything put it in here, and then basically explain what I want to do. Boom. So, another query, another prompt over here, I should say. I want to create a prediction model for stage one output measurement zero actual. Can you provide me with a function that engineer So, funny thing is you don't have to be grammatically correct over here. It doesn't even really make sense, but engineer features that could potentially improve the predictive performance based on all the other columns. And then this is really interesting. I say figure out a strategy that makes sense here given the name given the column names. Provide me with one function to add all the features to the data frame. So this again comes back to being really specific. What do you want? I want a function I want to input the data frame and what I want to get back a data frame with all the features. It knows what the target is. So don't just tell it, "Hey, create a function for feature engineering." Be specific. So let's see what we are getting over here. All right. So this is looking interesting. So it's looking at lags, it's looking at the window size. Let's see. What are we doing? Creates a copy, create lags for specified columns. Rolling window statistics for specified columns. Rolling mean, standard Okay, yeah. Yeah, so it's doing some basic time series feature engineering with rolling mean, standard deviations, min and max. So window size, lag features. Okay, so we have to define what kind of features we want to use. You can customize this list to include only specific columns. Okay. So let's just show you for the sake of demonstration how this will work if we use all of the columns, but we don't want to use the target column. And if I'm correct, so it goes for column in lagged features. And if I if I run this, so lag features, that will be everything. And then window size, what we can actually do and we can probably use GitHub Copilot for that. We say basically, so we're looping over and then I'm basically going to say pass if the column is this. So the target. Then we'll have to put in another if statement, and then if all right, and then we'll say continue. And I'm basically going to slightly change this. I'm going to set it up like this, and then indent this. So, if call not is the target, then it goes into this loop. So, basically means that we can pass in everything, and it will just skip over the loop. Lag features, window size 10 seconds. It's pretty short. I'm going to bump that up actually to a minute. And then, we're going to engineer everything. All right, and we're done. So, what do we have over here? So, now we have a lot So, how many Oh, okay, this is actually way too many columns probably. Ah, I see. So, what it's doing basically, we define the window size, and we have updated that basically to 60. And as you can see, it's looping over a range. So, it's not just using 60, it's looping over one up until 60, creating all of those lags. And then, for all of those lags, it's creating all of the statistics over here. So, that's why we have a shitload of columns right now. But, we can use another method to try and figure out what actually the best columns are. So, this is not even that bad, I would say. Now, let's just focus on feature selection. So, let's quickly export this data frame. So, we say data engineered. Sure, that is a proper name for now. So, we have the full data frame with all of the features in there, and then we can close it, and then basically come back to features again, create a new file, say feature selection.py. Okay, and we're good to go again. So now, let's ask chat GPT to create a feature selection model for us. All right, so we now have a lot of features. Create a function that takes the whole data frame and select the 10 best features with regards to the target. And again, I repeat what the target is. Return only the best features. So, what are we going to use? It's suggesting select K best from the scikit-learn library. Okay, so this is going to be interesting. All right, and we're done. So, we have a nice little function. Let's come over here. We already have pandas. Let's switch this up. Make sure we can import this. All right, and you can use this function to get the 10 best features for your data frame. Just pass your data frame and the target column. So, this is so awesome. So, we can put in our whole data frame, which has all the features, but also our target. And now, let's see. So, we have K over here, which is an int is at 10. So, here we can also increase or decrease the features that we want to get. And it says X Y. All right, let's do it. Also quite curious how long this will take. All right, and it finished. So, it worked, and we now have a data frame. So, if we look at the best features data frame, it returns only the best features. So, it's a selection of the data frame. And if I look at the columns, here we can see that it's actually all of the rolling statistics that we created using our feature engineering function. And just to create a correction, because I just figured this out, the function for feature engineering is creating a lags from one up until 60 in this case for all of the specified columns. And then also, it's creating the statistics for just the window size. So, not for all of the lags in between, but just the window size. Because I noticed that it's just the rolling mean, rolling maximum for these values over here. You can also see how it's using data from machine one, two, and three. So, that is also really nice. And also the output measurement. So, uh and since it's doing that, I also noticed a problem over here. Because we're cheating a little because we are feeding the target variable into the model as well, into the prediction. So, it's You can see how we have specified that it should skip if it's not the target, but we did not do that over here. So, let's also quickly come over here and add that line. So, we should now run everything again, basically. Uh let's see. All right. So, quick fix, quick check. We now, if we load the data frame with all of the features, basically, and we do a quick check where the name contains stage one output measurement zero, we only get the target variable. All right. So, that is a quick fix that I had to do, and now I have to run this again. All right. And we're done again. So, we have the new 10 best performing columns, and we can again see that we have a mix of machine one, two, and three. And now we also have the motor rampage in here. But again, it's all the rolling statistics. So, that is pretty interesting to see that the features that we've added are actually the ones that have the most predictive power with regards to measurement zero. All right. So, then we can actually start to use this subset of the data frame to create a model. So, let's first export it again. So, I'm going to say best features DF, and we're going to basically export that to the processed folder in this case. So, in the processed folder, I like to use or like to put data sets that are ready for modeling. So, let's put that in there. Best features. And let's copy and paste this, and then we can come over here to the models, and then train model. Do the DSI imports again, and say DF is read. Okay, boom. And now we have our data frame with all of the best features. And I forgot, I also have to add the target variable back. All right, and we're back. And this was also pretty interesting. So, I said, add the target variable, and then put the column name over here to the data frame, and then it suggested to use the PD concat. And that takes in a list of a data frame and another data frame or a series, and then specify axis equals one. But those are those really uh tricky situations where you really have to remember the syntax or GitHub Copilot can really help you with that. All right. So, with our new data frame, 11 parameters in total, 10 input variables, and one output variable, we are going to ask ChatGPT to set up a modeling framework basically for us to play around test with different models. Now, one thing that we have to keep in mind here is that we have not shifted the data in this case, and you must imagine that this is a process that all these chemicals, flows, etc. are going through. And what happens, for example, with machine one at this point doesn't directly affect the output on the other end of the machine, for example. So, there could be a delay. And depending on the kind of process that is at play, this could be a matter of seconds, minutes, or even hours before what is happening over here influences over things that are going on over here. But for now, let's assume that everything is lined up correctly and see what we can come up with. So again, let's just copy this whole data frame and ask ChatGPT to set this up. All right. So, this is the final data frame. Can you create a function that experiments with different models to predict measurement zero? Pick a selection of models that make the most sense in this case given the data and the column names. So, I'm pretty curious to see what it will come up with and we can already see that linear regression, random forest. Okay, so it's it's using some tree-based methods. That will be interesting. And then compare the models using the mean absolute error and the R2 score. Visualize the result. Please note this is a time series problem, so keep that in mind while creating chain train test split. So, this is something that I know from experience. So, if you use the random split when you are working on a time series problem, it will just randomly uh create that, of course. So, with a time series problem, you want to make a clear distinction between uh or I would should say you want to keep the data chronological. So, your train split should be before your test split. So, that's why I've added that as well. So, let's see what we got over here. So, it is coding away while I am talking to you and basically explaining my thought process and ChatGPT is just coding along. And this is also what I now that uh nowadays really do within my work. I really come up with something. I ask a query and then basically I go back with other stuff that I have to do and then wait for it or I continue to work on some code and then oh, once it's finished, I'll just copy and paste it. So, it's it like I've said, my workflow has completely changed over the past few weeks now that this is available because like look at all the code that it is providing. This would literally take like probably well over like half an hour for me to like set this up well and test it and then with the plots maybe even longer. Let's come back. We have the Uh let's see. We have the function. We have all the imports. So, let's actually move this up and we have to install XGBoost. So, we can do a quick pip install XGBoost over here. And let's also do a quick So, I know that the LightGBM model sometimes acts a bit weird on Mac, but let's see. Okay, and we're all good. So, if you're on Mac, the LightGBM, you should install it using conda. If you use pip, it can introduce some errors. So, data frame, all the dependencies, and let's make sure we store this function and let's quickly check it as well. So, using a test size of 20%. So, that is all good. And let's also see. Okay, so it's using Yeah, train test. It's using the index. So, it's not using the time series split or the standard train test split, but it's just using the index. So, that is all good. All right, makes sense. Target column, yeah, all good. Then we have all the models over here. We're using default settings, results, train and evaluate. Then results.append. Okay. This is looking really good. And then also it visualizes the data. This is pretty exciting. And then what I really like, it also just like provides you like Okay, here, this is how you run it. So, we can put that in and so it takes the data frame and the target. So, this is also a function that you can reuse for other code as well. So, it's already set up dynamically. All right, so let's see how this will work. Okay, first one. Looking not too good. Okay, okay. Results D, what's that? Probably messed that up. But okay, we have the predictions over here. Like I've said, they're not looking too good, but hey, this is our initial result. Okay, so these results are definitely not great. If we look at the mean absolute error, we can see that the simple linear regression model actually performs the best in this case. And you can see how in the beginning it kind of follows the trend. You do have to keep in mind that we are really zoomed in on the Y axis over here. So, this is actually pretty close, but then here there happened clearly happens something in the process and that is not captured in the data because it it isn't aware of that. So, what I tried to do is I came back to my feature selection script and increased the 10 features to 50. So, now we're using the 50 best performing features. And you can see how now there are also a lot of lags included in here. Okay, so let's see if we can capture the data better right now. So, let's run this again. Okay, that is not looking better. That is looking not so good. So, this is not really promising. Okay, again. So, the data wasn't really able or the models weren't really able to capture what is going on here to suddenly make measurement zero drop. Okay, quick little addition. So, I basically added all the basic features, so all the original columns as well because I am looking for the event that is happening over here. Basically, that makes the target variable lower all of a sudden. And it just seems that it is not really present in all of the data. But, by combining all of the basic features plus the best performing engineered features, we do get much better predictions. So now if I come over to the results, we see that we have some So the linear regression over here, I don't know what's it what it's doing over here that messes it up completely, but you can see that the mean absolute error for random forest, XGBoost, and LightGBM is starting to get smaller. We are still definitely not there. We have really weird uh high negative R scores, but you can see how it is somewhat starting to follow the data, and it is I must say it is a pretty noisy signal. So it's probably going to be hard to predict. But we are getting closer. But I had another look at the data, and I noticed that a lot of the values are very different in scale. So we have We have columns ranging in the order of 10, 100s, and also uh 1,000s. So I think what would what could work as well, probably not drastically, but could improve some of the performance if is if we add a scalar to this. So let's make another request. Simple add a scalar to the time series model comparison function, and we'll just do the standard scalar from the preprocessor. And now while we are waiting for the output from ChatGPT, I think it's also a good time to have a look at GitHub Copilot Labs and see what we can do with that. So I I've got a mixed results while using it, especially with the custom brush, which seems to me the most interesting one, which is basically similar to like ChatGPT where you can ask anything, and it will directly update it within your code, also considering the context of your code. So that is really neat. So I want to scale this data, and basically we have the function over here. So, I'm going to select that. And if you have not seen my last video, what you can do with it you can basically say like make this readable. So, let's see this is probably already pretty readable because it is already generated by ChatGPT. But let's see what it can come up with. So, it's basically processing over here. And now it has made the code a little more readable and we can also say we want to make this robust or you we can document it. So, let's see. So, now it's adding some comments over here, but it should in theory put this into a docstring to make it a little more nice. So, the kind it's not perfect yet. It should have recognized that this is a function and it should put it in a docstring, but let's try to make the same request using the custom brush. So, how it works is you select the code and then they do custom and then basically I say add a standard scaler in here to scale the data prior to training. Only scale the X. So, and what sometimes happens is that it will replace all of your code with just the new prompt. So, that is pretty weird. So, let me actually let me actually first copy this because also otherwise it's pretty annoying to to get back and now let's come over here and then say add the standard scaler. Okay, so let's see what it's doing. So, let's see if it can add it into this code. So, now okay. It added the scale. It added a boolean. Okay, so that is cool. If scale is true, we initialize the standard scaler and we scale the data and we only do it on the X variables. Okay, so that is awesome. I think that worked. So, you can see let's now see what ChatGPT has come up with. So, ChatGPT basically did the same thing. It converted the variables over here, but GitHub Copilot Labs added the nice feature to include it as a Boolean, yes or no. So, we can even we can set this to true by default, so we don't have to update it within our function over here. Okay, so this is pretty interesting. So, let's see if this still runs. So, okay. Linear regression still sucks. Okay, so this is now loading, but this was actually a really good example of how you can use GitHub Copilot Labs and the custom brush because this is what it's designed for, but like I've said, it's not bulletproof yet. It's not there yet, but it is working really well in some scenarios. So, I really how like how you can interact with your current code and then just ask it to input something or change something and it will do that. So, okay, the new results are in and like I've said, it did not affect the final results that much. We still don't really have a really or a really good We have a very bad model over here, but this also clearly illustrates why AI won't be replacing your job as a data professional anytime soon because you need that domain specific knowledge and this is a pretty hard setup right now since we are using data without much context as in we don't really know what the measurement zero is. We don't really know what happened over here, at least from the quick glance at the data that I've had. So, I'm also going to leave it at this for now because this video is getting pretty long. I will probably continue with this modeling problem and create future videos about this because we still have 14 other measurements or so that we can look into and I really want to explore the possibilities of ChatGPT and GitHub Copilot Labs more with regards to tackling a complete project like this. So, I could have just taken the Titanic data set or another data set that I am already familiar with and know that the output of the model will be good, will have a good accuracy, but that's not really representative of what it's like to work with data in the real world. So, this is actually far more common. You've actually no idea what's going on. You get a data set, you don't know how the variables are correlated, and you also really don't know if there is any predictive power in this data at all. Like I am I don't know whether this measurement zero actually can it can be predicted using the data that we have. So, I think this has been a really interesting approach to show you my AI-assisted workflow like this. I don't see a lot of people using it and explaining it like this on YouTube. It's far more pretty straightforward advice, but this should give you a complete overview of how I tackle my day-to-day projects currently. So, if you've been following along, then I would really appreciate it if you like this video and also of course subscribe to the channel so you don't miss future videos. And if you want to learn even more about how I use Chat GPT to tackle data science projects, then you should really check out this video next where I use another service to collect a web data set, and you can do this as well to basically do data science portfolio projects within a day or so. So, that's really awesome. Go check it out.