Naive Bayes from Scratch - Machine Learning Python

Key Takeaways

[Music] in this video I'll show how to implement naivebayes from scratch in Python and in the next video we'll use this implementation in a more practical example to classify emails into spam or not spam I won't do the like I won't focus on the derivation of naive bayes but I do feel I want to briefly mention some of the theory behind a base which come from the famous base 10 theorem and let's say we have some email which we'll call experience and we want to figure out how probable is it that this email is spam and not spam given this experience of previous emails so what we can learn is what is the probability of seeing a specific email so this evidence given that it is spam or not spam and this is the likelihood and then we have the prior which is basically what is the probability of an email being spam in general and lastly we'll have a marginal likelihood which is a normalization constant and we'll ignore this part because it won't impact the resulting classifier that we'll have let's start with some skeleton code of how our Navy base is gonna look like so we're first gonna import numpy as NP we're going to do class naive bayes and we're gonna have an init function and here we're gonna just send in our data x and y so x will be sort of the they're not all the training examples and then all of the features in a matrix and y will be the target class for each training example we'll have a fit function which sort of yeah this is sort of the training phase of naivebayes and then we'll have a predict and then again we'll send in so we'll train it on some data will predict on some data this could be the same or it could be if you separate it into a training and a test set and then we'll have a density function and we're gonna send in so the data that we want and then we're gonna send in computed mean and variance Sigma and what we're gonna do here is we're gonna calculate probability from Gaussian density function so normally distributed so that the likelihood is something we can model from different distribution which also gives us different naive Bayes the one-wheel model that one will implement is the Gaussian naive Bayes so we're assuming here that the probability of seeing a specific email given that it is spam or not spam is normally distributed so we're gonna use a Gaussian density function and from this so from this assumption of being normally distributed we can do maximum likelihood to find sort of the optimal mean and variance parameters to fit our our data of emails and I just want to do a quick clarification as well why it's called leave base which is the assumption that if you know that email is spam and let's say you know the word bar is an email that won't impact the probability of money being in the email you know which doesn't hold true in general but it's an simplifying assumption and it still gives pretty good results in practice so as I said we're gonna use the Gaussian density function and we're gonna use a trick here so we can take the density function just from Wikipedia and I'll show it on the screen and then we'll perform a a log operation which makes the the numeric it makes it more numerically stable and this log is a monotone function it won't impact the the resulting probabilities the maximum will still be the maximum after taking the log we're gonna do self dot num features I'm just gonna write this out this is just from the the density function of of a multivariate Gaussian I do want to mention here that the implementation isn't identical to the density function that you see on the screen first we have performed a log operation to make it more than miracle II stable but also simplified a few of the calculations with the naive Bayes assumption so the the Gaussian density function is normally a multiplicative so you normally multiply the constant with the probs with the probabilities but but since we're taking the log that makes the product into a sum and that's why we're taking we're subtracting bar by probs here so I'm using self dot eps here let's say and we also using self to numb features let's just create those in our init so 7 on features and self dot examples is let's say it's actually self dot no examples first is X dot shape so X is the rows are the number of examples and the columns are the number of features and then we're going to do sort of num classes we're gonna do Len length of NP dodge unique of Y so let's say that y is you know one two three one two three four something like that and be unique just finds all of those unique values which in this case is just one two three four and it just takes the length of those but let's say we would have four four then it would still be just one two three four because these two wouldn't be unique in this set then we're gonna use Celtic apps just for numerical stability let's just say 110 times -6 then in the in the fifth function we're gonna do and we want to figure out sort of the the mean and the variance step we're going to send into the density function and we're also we also want to compute the the prior here so let's do self taught the classes mean self dot classes var self-doubt classes prior and then let's iterate through all of the classes that we have so we have four last let's just do C in range of self dot num classes so what I'm saying here also is that the classes are starting from zero so we start first class is zero second is 1 2 etc and we're just gonna take what are the examples that we have that is a from a specific class from our data set so we can do X and we can do y equal equals C so this will pick out the training examples the emails that is from a specific class in this case let's say all the emails that are non spam and then all the emails that are spam and we're just going to compute self that classes mean for this specific class so we'll do string of class in this dictionary and then we're going to do MP dot mean X C and then X is zero so sort of why we're doing this comes from the theory of doing maximum likelihood which will not derive but will rather just use those results and then also classes dot the variance in two variants same string of C is NP dot variance of X see X is zero as well and then we're going to compute the classes prior and so this will just be what what are the amount of examples that is from the specific class so we'll just do X see that shape of zero and then we'll divide it by the entire the number of examples that we have in total so we can actually do self Nam examples okay that's it for the fit function then what we want to do is we want to do the predict we're gonna do props you're just going to be emptied out zeroes of self that numb examples this is the shape and then self that numb classes so we want to have sort of a probability for each example being each of the classes so if we have an email we want to know what is the probability of it being being spammed what is the probability of it being not spam since we in this case we just have two classes I mean if we would have email this is a general implementation you can send in any number of classes then we're going to do for C in range of self num classes we're going to do the prior going to be self-taught classes prior of that specific class then we're going to compute the props of that specific class and we're gonna do that with the density function so we're going to send in X we're going to send himself that classes mean self-doubt classes variance king of c and then we're going to add so this computed for this specific class we're gonna compute we're gonna store them rather in the props that we define here so props of all examples come out just that specific class we're gonna set it to prop c plus the log of the prior right at in the beginning i show that this probability is actually the problem the likelihood times the prior but since we're doing log we convert it into a sum so this is the result for that and then at the absolute end we want to return and P dot art max of probs come up 1 and here we're taking the axis 1 because we want to have for each training example we want to have what is the the class that it predicts the highest then I notice here that we've we need to send in Y here and we also so I used variance but here I said var so let's do variance so it's all the same and then let's just do this for some I have some example data so let's just do if name it goes main so let's start with loading our data and I have below the data in a separate folder example data you can download the exact same data if you want to follow what I'm doing from from my github which is in the description but really we're just gonna NP that use NP that load text and we're just going to specify example data and then we're gonna data.txt we're gonna use the limiter is a comma and then we're gonna do why is MP dot load txt the same thing and then targets a txt and then the delimit we're not gonna have any limiter in this case and then we're just gonna subtract by one because we want the classes to start from 0 and then 1 2 etc now it's starting from one and two three etc so we will just subtract one and let's actually try to - now yes let's do now you bathe let's just sending X&Y and without v of X and then my prediction will be and without predict of X so this is sort of the training phase this is sort of the prediction phase and normally I guess you could have a training data and a test data in this case we just do on the same data let's try and run this we're probably gonna get some errors yeah as usual we need to have x and y here and then we need to have I guess just X there yeah all right so yeah okay so there's a mistake here there should not be brackets this should be just a normal practice this and yeah so the same here here and then that should be it I think let's see if it runs now okay great let's let's first actually print just what is the data that we have what is the shape of it so we have 90 90 90 training examples or a point in this say in this case we have two features so they're in the XY plane and and we just have targets of 90 examples right well class has a single one so let's we can print Y for example and so we see that there are three classes and then after the prediction let's do some of Y prediction equals y and then let's do divided by X dot shape of zero so we're sort of getting an accuracy urghhh we can see that it's an 90% accuracy so the data has some noise I'll show how it looks like on the screen but there's some noise so 90% seems reasonable and yeah so that's an exam in the next video we're gonna use this implementation to classify emails into spam and non-spam and yeah thank you so much for watching your video if you have any questions leave them below and [Music]