Python Environment Setup for Machine Learning

Key Takeaways

hello and welcome to this video on how to set up a python environment specifically for machine learning so this is often an overlooked part of machine learning and there's not that many tutorials out there on how to do this properly so i thought it'd be a good idea to just go through this and explain exactly how i set up my environment so you can see here we have jupiter and first thing you might notice is that i have these different environments you have the default python 3 base environment and then i also have this gcp which i use for the cloud and then i also have this one which is my machine learning environment now the difference between each of these is the machine learning environment specifically has packages in python for machine learning like tensorflow pytorch transformers pandas numpy it has all of those packages but nothing else so there's no excess baggage if you like so i wanted to just go through and explain how to actually set this up so we're going to close this jupiter notebook i'm going to open this new anaconda prompt here so i'm assuming that you have already installed python and that you are using the anaconda distribution so if you are not using this you can download it over from here you can head over to anaconda.com and you just click on products and individual edition over here and just download okay so the installation for anaconda is pretty simple if you're on windows it's a little different if you're on linux and i i don't know how it is on mac but generally it's it's pretty straightforward and if you do need any help with it you can just you can find out quite quickly so once we have that installed we want to go over to our anaconda prompt and to make sure that we have installed it correctly we just want to type python v and this will show us the version of python uh that we have so i'm at the moment using python 3.8.3 just make that a little bigger and okay if that works that's good so at the moment we're using the core based environment that you can see here and that is just the default environment that gets installed whenever you install the anaconda distribution but what we want to do is actually create a new environment which is our machine learning environment to do that we use the syntax like this so we condo create conda is just referring to anaconda or name so you can also write this as name or again with n and then you want to enter your environment name here and then you would also write python and your python version and at the end of that you would also type anaconda so for us i'm going to use a [Music] environment name of mln like that and i also want to be installing python 3.8 and that should be everything so we'll just enter and now python will work through and actually install that so i've already installed a mlm before but i uninstalled it so it's coming up this warning saying it already exists but i'm going to continue creating the environment because i want to reinstall it so put yes you shouldn't see that on yours and then this will take a little bit of time just to get everything together okay so now we are just showing a list all the packages that will be installed um so we just want to accept that so press yes and enter now we'll go ahead and install all of those okay so everything is set up now we can switch over to our new environment so at the moment we're in base we can switch over to our new environment with conda activate and the environment name which in our case is mln so let's go ahead and do that and now you can see that the name here is switched to mln which is our new environment now we just need to install our machine learning packages so we're going to go ahead and install the basics so we have pandas and map plotlib we're going to install both of those with a condensole so we have two options here we have conda or pip to install our packages generally conda will most likely integrate with your environment better so it's usually a good idea to try it at first if that doesn't work then try pip install so we'll go ahead and condense soul we're gonna do pandas and matplotlib then add anything else here that you feel that you might also need but this is all we're gonna go with so it's a good point noting that we also need numpy but numpy is included as a dependency of pandas so we don't need to explicitly mention numpy here and that will go ahead and ask us for permission to install the packages that it finds we click yes and then we go ahead with the installation again now we can go ahead and install tensorflow transformers and pi torch which are all machine learning frameworks so tensorflow we can install it quite easily all we need to do is click on the install tensorflow okay so now we have the yes or no from tensorflow okay so tensorflow is now installed so we can now go ahead and install transformers so transformers we are going to use pip because condor doesn't recognize the most recent versions of the transformers library at the time of recording at least so we have to use pip to get the most recent versions so we pip install transformers there we go and finally we have pie torch which is slightly more complex but we make it quite easy by just taking a look at the pie torch start locally guide which you can find here so pytorch.org gets started locally and all we do is we come down to the start locally bit we select our pytorch build so this is the stable release and this is like a beta release which gets released more often but it's more like to have bugs on errors in so i think most people will probably want to avoid this you can choose your os so for me to windows package manager so that is condo you can also use pip but i would recommend that conda because it will install the dependencies we need as well we're using python and then this bottom one here refers to cuda so we used cuda as the gpu acceleration library so essentially with this if you have a nvidia gpu cuda lets you use it speed up any machine learning tests that you have um in either pi touch or tensorflow so you can read tensorflow's gpu setup guide if you do have a gpu this is quite useful so you just you head on down to the bottom here or if you if you're on linux as this guide is always quite useful and then we have the windows setup here so all you need to do is install all of these which is reasonably straightforward but there are a lot a lot of good guides out there if you do need help with it and then you just head on down and set your paths so that tensorflow slash pi touch you can actually see cuda another useful guide as well is this nvidia cuda installation guide which can be quite useful as well now i would recommend using cuda 10.2 at the time recording so unless you are using the latest rtx 30 series so that is the nvidia geforce rtx 390 3080 and i think it's 30 70. so the support flows is a little bit sketchy at the moment and you will actually need cuda 11 alongside the nightly builds of pytorch and tensorflow so this is what i mentioned over over here that's a little bit more difficult and i'm i'm not going to be covering that here but again there are a lot of good guides out there if you do need help with it so if you don't have a gpu or you just don't care about gpu acceleration you just click none and it will change the command down here which we'll be using for our installation so i'll be using this command here so we're doing a conda install and then we have a few packages not just pi torch here pie torch torch vision torch audio cuda toolkit and first we are using a 10.2 and then we it's our channel to pi torch as well now we go ahead and install that so just select yes again and now that is our environment completely set up so all we need to do now is actually add this environment to jupiter so remember at the salt we had that little box and we had python 3 gcp and ml so we're going to add a new one called ml environment so to do that we need to install ipad kernel and with that ipi kernel we are going to install our new environment so we do that by specifying the name of it here mln and then we also want to set the display name so this is the name that we will see when we enter into jupiter lab and we can have that box so this can be anything you want so for me i'm just going to put ml environment let me just run that again okay so that is ready and now we can just go ahead switch back to our base environment which is our default environment and now just open up jupiter lab and we can see here we now have this other ml environment and this is the one that we just created so that is it for this short video i hope it's been useful and i will see you again in the next one thanks for watching bye