Natural Language Processing|Stemming

Key Takeaways

okay so this is my stemming code okay and if you remember the tokenization code is something here I'll upload this in the GitHub you don't have to worry about it but with respect to stemming I'll be using the same paragraph that I had actually discussed while doing the tokenization that is a speech from the famous former president of India that is apj Dr apj Abdul Kalam right so if if I am downloading first of all I'll download nltk okay first of all I'm let me just read restart my kernel because this was the part from the this all variables that I see over here is basically created from the previous tokenization so let me just restart it and once I restart it I'll again run the code completely Al together for the from from beginning to the last now for nltk first of all I going to import the nltk library which we are going to use every time for the text pre-processing once I import this there are two things that we have to discuss about one is stemming and one One More Concept is something called as stop words so stemming will basically be done by a class that is present inside nltk do stem and I'm going to import this Library which is called as pter stemmer and I'm also going to import one more Library which is called as stop wordss and stop wordss is actually present inside nltk doc Corpus so let me just execute this two more lines I'll just explain you what is stop wordss so first of all I'm going to import the stem in the library that is used for stemming and the library which actually includes the stop words I'll just explain you about stop words then I'll just execute this paragraph okay I'll just execute this paragraph now this is my paragraph over here just let it get executed fine it is it is got executed now first of all I'm going to explain you why I'm using the stop words in this particular paragraph you you will see various kind of words something like this off from right are the and our of them this kind of words are repeated again and again throughout this particular paragraph and this word does not actually play a better role actually play a better role when we are when we are solving different kind of applications with respect to positive and negative sentiment analysis because of the them does not actually specify a much value in that particular sentences right so stop words helps us to remove this kind of words so once we apply stop words in this particular paragraph it will remove all these different kind of words that we have words that actually does not put much value on this particular scenario or application but there are some application where this kind of stop words do make sense which we'll be discussing later on but for most of the scenarios is we do apply stop words and try to remove this kind of words okay but before that again I'm going to what first the step First Step that I always told you that I have to convert this paragraph into sentences right for converting this paragraph into sentences I'll be using a function which is called as sentore tokenize so once I execute this you can see that I'm having all my sentences over here it'll take some time because it is a bigger paragraph and I have you can see that 31 sentences that you can see over here right now before applying then then then the next thing is that I have imported a library which is called as pter stemmer which was for stemming purpose right so this pter stemmer I'm going to initialize it and I'm going to create an object of that that is stemmer I've renamed it to stemmer now what I'm going to do is that for each and every sentences I'm going to apply and I'm going to remove those words that are present in the stop word if you want to see the list of stop words what you have to do is that you just go to this console and just execute this particular statement that is stop wordswords and I want all the stop wordss with respect to the English language once I execute it you can see that there will be a huge list of stop Words which actually does not put much value in a paragraph when we are doing some applications like positive and negative sentiment analysis so you can see some words like I me myself we our ours ourselves you you and all these words are basically stop words right best thing about stop word is that you also have stop wordss in different different languages right so if if I go and execute the same thing with different language like French I'll write it as French let's see whether we have any stop wordss for French if I just execute it yes we do have some stop words of French right similarly with respect to this I may also have some stop words with different different languages like Spanish or just let me write German okay if I execute with respect to German um you can see that there are also words with respect to German right so there are lot of languages this stop what actually you know it is present and these all are present in inside analytic Library so this analytical Library can be handy when we are doing text processing with respect to the other other languages also so first of all I have actually imported stemming now my next step will be that whichever words whichever words are actually present in the stop words I'm going to remove those words from the sentences right that is the first step then after removing those stop words I'm going to do the stemming for each and every word by using this pter stemmer or the stemmer object so it'll be a very interesting code you just see the code focus on this particular line foring what I'm writing is that I'm writing I'm putting a for loop on all the sentences so I'm putting a for loop on range of length of sentences my range of length of sentences is around 31 so basically my range will be between 0 to 31 for each sentence first of all I'll do a word tokenize word tokenize basically will convert all the sentences into words so this words will be a list of words from each and every sentences right from if I just take the first sentence it will be one 2 2 3 4 five six words right then after that what I will do is that I will just write a for Loop saying that for words in Words for word in words so basically this words is the list of words for each and every word I'm going to you know iterate and I'll write a con I'll write a condition which says that if word not in set stop words. wordss of English the reason why I have used set set actually help us to take the only the unique stop words in the English language so that unique set of words stop words from the English language will be compared whether that word belongs to this or not if it does not belong then only it is going to go and do the stemming and stemming will be done by just writing this object that is stemmer okay dot stem and then I'll put the word over here so if there is a stemming to be applied in this word the word will get stem right and this this this whole thing is basically called as a list comprehension right now you see in list comprehension I'm just basically putting a for Loop and then I put the condition if that word belongs to the stop words or not if it belongs to the stop words I'm going to remove it otherwise I'm going to do the stemming process on that particular word after I get the list of words I'm going to join that words and convert back into the sentence right so I'm first of all what I'm doing I'm taking the sentences I'm tokenizing into words then I'm applying the stop words on that word if that stop words uh if if the word is present in this stop word list then what I'm going to do I'm not going to take that word instead if it is not present I'm going to take that I'm going to apply the stemming onto that and finally I'm going to join that into sentencing so once I execute this you can see that I think it'll give you an error because I have not executed this two line of code so let me just execute it I think I have executed the sentence undor tokenize but I have not executed the port stemmer let me create an object with respect to Port stemmer so stemmer is the object that is created I'm going to execute this three line for the follow and finally you'll be able to see that my sentences will now not have any stop words so you can see that I3 Vision India in 3,000 year history see history has been converted into h s o r i so stemming has happened over here people has been converted into p o l and similarly invade has get got converted into in inv vad right capture also has got converted into so basically all the stemming has been applied to this right so this this this if you see all the sentences you can see all the changes over there now let me just go back to my PPD let me discuss the problem with respect to stemming the main problem with respect to stemming is basically it produces intermediate representation of the word which may not have any meaning so most of the word that is getting created after stemming it is not actually having any meaning so some of the example is intelligent gets converted to intelligen intelligent does not have a good or proper meaning right similarly final gets converted into final here also we saw some examples if I go and see the sentences the history has got converted into history people like p a p o p o PL similarly all the other things you can see over here conquer has got converted to C and Q so in order to overcome uh disadvantages we basically have another type of uh stemming technique which is called as liation which we are going to discuss in the next video I hope you liked this video please share subscribe and share with all your friends I'll see you all thank you