Extracting Value from Data

Key Takeaways

so hello and welcome everyone uh to another session in the data R series so we are thrilled to be here you this evening for a session full of action-packed learning I am a sambhav Chan part of the data science team at analytics Vidya for those who have joined us for the uh very first time uh brief introduction of the data R sessions is the data Arts is a series of webinars conducted by analytics Vidya and led up by top industry experts uh it is a fun way to understand the concepts of data science from the leading players in the data Tech domain and as the name suggests it is one art dedicated to data so we are hopeful that these sessions are going to be great source of enrichment and value adding for our community members so now on to our session today which is uh our total uh our today's session on extracting values from uh data so in that data V covers in this data are we cover like uh topics like types of data structured unstructured data basic data checks variable types q and as variants null value Etc outlier detection and its treatment data transforming and uh redundant variables reducing data complicity data sets examples across different Industries and their use cases data wrangling Concepts Etc so I have uh you all are excited to attend this data with us so before we kick off uh things off and I hand it you to uh the today's RSP card so a quick recap of uh housekeeping out on items like we are recording these sessions and we will make the recording available in a few days on our YouTube channel so please uh also uh you all can use uh the Q a session for asking any questions you might have during the sessions as well and we will do our best to answer them as the data progresses or towards the end so also we will share a poll about the feedback of the session towards the end of the session so which I requested you all to kindly fill up so now on to our speaker in this session of data R we have uh Varun behal with us Varun is currently working as data science engineer with Adobe in Bengaluru he has more than five years of experience in data science profession and has worked with various kinds of data covering Industries like Telco e-commerce Pharma retail Muslim and web analytics so he also has Keen interest in research side of like machine learning with NLP forecasting segmentation and recommendations being areas where he contributed significantly for enhancing the existing scope of traditional modeling techniques so that's a uh about our today's speaker so over to you Varun the virtual stage is all yours I'm sorry um hi everyone and thank you for taking time out for this session um as the title says we are talking we're going to talk about the data primarily and as a topic says it's called extracting value from data so primarily covering all the aspects of data from understanding what all variables it it has how to understand what kind of variable it is what are the properties of the variable how do we understand uh um how to deal with different kind of data sets um it also caters to the problems of outlier detection and treatment which is primarily the most important aspects when we are building any machine learning model data transformation and processing is also a key aspect where we want to understand what's the dimensionality of the data and what transformation will help us in structuring it in a way that is suitable for a best fit model then briefly we'll talk about data or angling operations that you can in day to day life if you go on to uh become pursue data science field that is one basic step that you would need and in any particular language that you code in SQL or python or R these are the basic aspects that we are going to cover and lastly we're going to talk about some data sets example and along with that if there are any questions answer we I'll be happy to answer those as well being said that let's begin with the session and the first topic of this session is type of types of data um if we if you look at the various types of data we can further classify them into unstructured and structured format um primarily the structured format is where which is already in a format in which you don't need to do much of a cleaning it is already in a form which is a easy reusable form or you can do directly use that to build a model whereas unstructured data is anything that you can be scraping from different websites it could be uh scrape through an API or you are maybe translating as particular word document or a PDF document or something so every data when we talk about the source of the data I've given four classification to understand how an unstructured data and structure data are different so unstructured data is an unorganized form it's obviously a qualitative data we do not have categorizing to a quantitative form so the capturing process of the data can happen through the web session log whenever we are traversing any website the cookies track what our behavior is on the website click stream data on the website transaction data all this when it capture gets captured in the raw form it's primarily very unstructured and there is hardly what we can do to create any value out of this so it requires a lot of pre-processing a lot of um sophisticated techniques are on top of that apply to properly create into a structured data form now in terms of the storage the two data uh the two types of the data one can be actually stored in a relation database where you are having a SQL database it could be an RW uh redshift environment or it could be Hadoop environment where these relations databases are stored there are well defined columns and they are attribute it whether they are integer column or an American column or a categorical column whereas when we talk about unstructured data no such classification is available that's why they're kind of stored in data lakes in current world we primarily use large amount of unstructured data and we are stored in S3 buckets or a large pool of servers that are there in-house to any company in terms of operationability of these two kind of data unstructured data is non-scalable by non-scalable I mean if you want to process it the way you process the structured data that that means if you want to query this particular data or you want to apply some data wrangling operation that will be a fairly difficult task the search operation if you want to search any particular thing all these particular aspects are very difficult to address when we are talking about this now what differentiate between like when I am saying structured data is scalable what makes it scalable is the way it is structured in a relational database that these are usually partition or distributed in a way that there is a particular primary key that will help you in differentiating what is the level of the data so if a data is leveled at a set of columns then you can fairly classify it into a structured data whereas unstructured data will not have any primary key or anything that helps us in identifying what could be the level of this data that is where the transformation is done on unstructured data to make it in the format that is usable now unstructured data requires a lot of sophisticated methods like NLP natural language processing to actually make sense out of the textual fields or the image or anything that has uh been directly in the Raw format been scraped out and it requires a lot of cleaning to particularly create some columns out of it some variables out of it that would actually make sense so that is a gist of how we classify our uh two different types of data whereas when we finally transformed ourselves into a structured data format there are certain categories of column that would exist in uh so in the real industry if you look at any kind of data set there will be different kind of variables that would be available I would further classify these uh variables into a nominal data an ordinal data text Fields discrete which is could be your category continuous data and date timestamp so in any real-time data you will always find these six components available in some of the other formats and I'm talking about nominal data it caters to anything that is related to gender ethnicity area hair color locations and all those so kind of it is also a categorical field but it has a factual reference to it available whereas when I'm to differentiate it between the discrete or categories it is the field that is created by someone it's it's a derived particular categorical variable so if we could even consider our customer names customer ID or session IDs as a part of a discrete Factor variable even our Channel names and product names are also part of it because in the Raw format you don't need usually find these attributes whereas we transform them and derived them into a format that is usable continuous data is all a numerical fields or float variables that we usually deal in our day-to-day life it could deal from uh salary Revenue number of order purchases that can be happening on a website so every continuous field that is in the integer or could we have decimal points if it's related to salary revenue or any cost related attributes even ratios that we usually see in our day-to-day daily life that also come under the category of continuous data we would always see if we are dealing with the time series data then there are date and time stamps even for non-time series data there are usually date annotations that are given to understand on which they that particular row was edited in the data or it was appended in the data so these kind of time stamp field helps us in understanding how the data is being captured what was the uh chronological order in which the data has been captured and it helps us in sorting in all the uh further exercises that we do on the data there could be a lot of text Fields when we primarily talk about data that caters to our is built based on surveys or the website that is great this data is in a very raw format could be customer feedback and reviews so these kind of uh Fields actually increase the size of the data but they are clearly important as the information in them is in a very raw format and there can be a lot of use cases that can be developed when we clean these five particular fields or create a structure output from these particular Fields now ordinal data is any it could be categorical or it could be integer but it is something that you can sort it so if if we have let's say given age of a particular uh patients in a particular data that can be sorted based on um any even ascending or descending order so these are the kind of data aspects that you can sort it could be even in a categorical manner that yourself but there is a property to which based on which you can sort them moving further uh wanted to touch base on what are the necessary data checks and the cleaning step that you should do so this is where your exploratory data analysis starts so let's say you have a huge chunk of data and you want to understand what is the approach that I want to follow in understanding or reducing uh the right characteristics of the data so when we talk about necessary data checks we should easily first classify them into a univariate test and a multivariate test on this screen you will currently see a univariate analysis what I have done here is I've uh in which you first check what are the data types of the different columns that are available so in a particular data scenario you will have a lot of columns in your data and you would like to understand what data type it has because understanding the data type tells you what is the next step that I want to do in understanding the statistics of this particular field so if there is a numerical data you will come across um variables like that could be data having data type of m64 float num64 or similar integer in categorical we might come across class objective factors all these tie back to only categorical Fields numerical Fields could have properties like ink float or non numerical six four now these are the types that you will obviously see when you are maybe coding in python or understanding whether uh to seeing the summary of that particular field whether it falls into integer category or a float category even in SQL the similar approach can be dealt with and you can understand what the data type is now let's say we understand the data type is numerical or it's a continuous data so you can go through those various statistics and that is the first step I would recommend everyone should look into these are the basic stats that you must be familiar from mathematics checking the mean of the particular column checking the Min and Max of the particular column checking the median of that particular column and similarly standard deviation and percentile values and there are other two important aspects to uh other than the standard uh statistics which is kurtosis and skewness it helps us in understanding what kind of bias the data has or whether it has a high variance or it is normally distributed similarly when we talk about categorical data we talk about distinct counts that are there of the number of categories that are there how many missing values that could be there and the frequency count across category so let's say if there is a category of a hair color of uh in of the citizen in a particular area there could be a lot of people with brown hair there could be a lot of people with back hair going to people with uh you know of color their hairs and different colors so that frequency distribution tells you that what which category is prominent in which category is least prominent and the visual checks that usually will help you in understanding these particular universities primarily is box plus and histogram they are very sophisticated um visualization approaches and available but the standard if you are starting with any analysis box plot and histogram gives you enough information in understanding a continuous field and that will also tell you what kind of bias exists in a data whether it is rightly skewed or right inclined towards the values or left inclined towards the values now what are the cleaning ways that we could initially talk about when we have just seen the data these are the first steps that everyone should be taking care of and in in the data set so always check for the duplicate values in a particular row so when you get an overall data you should always deduce that particular data in understanding what level what is the right level of the data if there will be duplicated records you will never be able to identify the right level of the data and identifying the right level of the data is a key approach that you do in even in your further exercises over angling even if you are merging with another data set you need to have the right level in place then only you can merge with any other data set talking about the null values null values kind of exist in all of the real life uh data examples that I have come across and sometimes null values make sense and sometimes they don't so there is always a business sense to be applied when you're looking at what should be the right approach to deal with the null values so if the null value is in a particular column in not making sense it should be imputed either with a mean value or a median value we'll be further talking about this particular problem statement then they we how to cater to the null values now date shoot date field that are existing in the data sometimes get classified in the numerical format sometimes in a categorical format or sometimes as text so we should always make sure that we convert into them into the right timestamp and day field because that is important whenever we are doing any sorting exercise talking about a textual or a categorical data we've always seen there are some kind of typos that might exist in the data and there is a need to deal with those typos so there are a lot of sophisticated approaches as euclidean distance and fuzzy matching available that you can apply but even before that you should check for any white space any unwanted text any special connected rights at the rate hash underscore slashes you should be not having this in your data when you are cleaning them there could be uh fields in which you could have some suffix or prefix added to the data and you might see that as an redundant information and that is also an information that you can obviously remove from your data set so this is a kind of uh approach that I would recommend everyone to follow when they are starting with any data check exercise and then what was the first cleaning step they had to apply foreign data checks and cleaning exercise here I am going to cover what are the multivariate ways of checking whether a particular variable in the data is a right fit whether we should be keeping it in our data set does it make sense to keep that in when you are running a machine learning model how do you come to a conclusion that this particular variable is important for us or not now these there are kind of different kind of statistical tests that are available and you can obviously apply an understanding this you can obviously plot between two variables a scatter plot to understand what is the relation between them there can be a correlation test done between variables to understand whether a particular variable should be kept or the other variable should be kept now in terms of the test that you can apply to and that will also help you in cleaning out unnecessary variables out of your data one is the T statistic test now this t-test is not a normal tease that's it's the t-test of Independence when I'm saying Independence it basically checks whether two variables are associated with or not and when we talk about Association we usually talk about Association in terms of means of the variable or a median of a variable so if the two if you are comparing an H column to a revenue column and I want to understand whether they are very Associated to each other then I will take a call whether to keep a revenue field as a continuous variable in it or age field whichever is making more sense to me in terms of the use case of the problem now if the T statistic says that the both like we we are basically testing our null hypothesis here whenever we I'm talking about a statistical test here it is about this proving or approving your null hypothesis so if a null hypothesis is not disapproved or it holds true that means both the fields are similar to each other and you might have to drop one of them or you could do some feature engineering to combine them to create one out of them now if I'm talking about t statistic that is primarily used for comparing two variables and those should be continuous by continuous I'm saying it should be in a numerical format you can do standardization or normalization but even if you are checking in the Raw format it will give you a spec a basic interpretation of whether uh which particular variable to keep now every statistical test gives you an output in a p-value format if someone is not familiar with p-value p-value is the point in a normal distribution which helps you understand whether it falls in a critical region or not now if a p-value of a particular test comes out to be less than 0.05 then we can fairly say the both the variables can we can keep and they are significantly different from each other similarly when I'm talking about categorical test I would be uh I'm talking about categorical variables I need a different test other than the t-test chi Square test of Independence is primarily applied in understanding whether two variables or are associated to each other in any way again chi-square also will generate a p-value for you it's there is a chi score also Associated to it so you can take a call as per whether you want to understand for uh to as per P value or Chi Square value whether the two both the categorical Fields make sense in the data or whether we need to keep one and eliminate the other one I've also added a table here of level of measurement which basically tells you if what kind of variable if it's a nominal variable and if you want to compare it with another sample and what kind of sample it is what kind of test you can apply so these are the kind of tests that you can always have access to um and you can obviously apply in any of the to tools that you are familiar with you can do these exercises even on an Excel even python always as in modules available for these particular things and you can even create the formulas for these and can apply the same statistic and get an output on a SQL platform as well now moving forward what are the ways in which you would be able to uninterpret these test statistics we talked about the p-value but there are other two key aspects in which helps us in understanding whether our data is fit for building a model or not now here I want to focus on a bias bias and variance trade-off and you will always keep on hearing these terms when you are starting your data science Journey or even you are solving real life data science problem bias and variants are the two killers uh which can hamper your model accuracy it can cause Misfit of your model so how do you first get an intuition that your model might be skewed or might be biased in a particular direction so skewness in kurtosis are two kind of tests that people primarily use in understanding whether there is a high variance in the date or high price there can be other approaches in solving whether your mean is closer to the 99th percentile value of that particular field or mean is closer to the first or 25th percentile value both these ways are easy identify whether the data is skewed on One Direction or not here I have shown two example when the skewness is less than one the data seems to be right least cubed so that is an indicator that there is some bias existing in the data and we might have to cap or do some training of those lower values or those outlier values to create a normal distribution because normal distribution is kind of a right format that you would need to build any traditional machine learning model and that is one of the basic Assumption of our linear and logistic regression tool talking about skewness if it is higher than one then we might say that the data is left skewed and then again it requires a similar way of treatment and this skewness basically is an indicator that there is some kind of bias available in the data so if I want to understand what is my variance in my data I could basically plot a scatter plot and understand if it is not falling and the values are scattered across the entire X and Y axis then probably there is a very heteroskedastic data which is basically a term given to an high variance or the data that when checked with a dependent variable shows you an unfit plot the kurtosis is a measure if it is closer to zero it is basically tells you the data is centrally aligned or it is symmetry and when the ketosis value Falls away from zero it is basically due to some kind of variance that is available in the data and that is not good when you are fitting any model now what are the possible challenges that you might face or with the possible outcomes that might see when you are seeing High bias or high variance in the data Whenever there is a high virus your model will tend to overfit because these are the noise points that a model picks up and tends to understand this as the right value and when you over whenever you are doing any sort of prediction say regression or classification you will always see your model on overfitting when you are testing it whenever there's a high bias in the data the model tends to underfit there is a bias towards some set of values and when you in the test data you will not see those kind of values the model will under fit and it will never be able to predict those values correctly so the optimal good balance is when we have a data set in which or a variable which is treated for low bias and low variance so there are a lot of approaches that we're going to further talk about in the upcoming slide on how to treat this low bias and low variance and primarily what we are going to talk here about is the outly treatment of every of these variables which kind of helps us in reducing the bias and variance in the data as well now outlier detection and treatment is basically a treatment step that that outlier could exist in the data that could be based based on the capturing issue this could be a data outage could be a data Spike could be redundancy in the data or could very well be in a human error as well all these possible scenarios can lead to an outlier in your data but there is a utmost need to understand what these outliers are there has to be a proper way in understanding how to identify and whether or not we should be eliminating these outliers whenever we are talking about outlier detection it is a kind of a sensitive approach we should be following to it because there are certain Outlets that might make sense as per the business logic so if this caters to a business value if that particular lower or a higher value that we see in the data if it still has some justification as per the business there has to be an intent to keep it but there has to be some kind of treatment to be applied to that particular outlier as well so that it makes sense in the overall sense of that particular variable so there are various ways of checking an outlier and but I would be fairly talking in concising them over the two approaches which are fairly easy to understand and easy to implement as well so first one is the box plot or the interquartile range whenever we distribute our data set uh in a box plot approach it qualifies the data in a confidence interval range the confidence interval range is basically your what is the what is the first quartile range to the third quarter range in in simpler terms what is the 25th percentile value in the 75th percentile value as per what you get in your data you can obviously extend your percentile value some might like to use the fifth percentile in the 95th percentile whenever we are talking about a t-test or anything we usually consider fifth the percentile value and the 95th percentile value as the range of a confidence interval whereas when we are talking about IQR we usually keep it the capping at q1 Q3 which is nothing but 25th and 75. again when I'm talking about z-score it's also another way of checking your outlines here we are checking whether it falls in a critical zone or not if you look at the below chart I am showing the normal distribution plot and there is a critical region that basically hints that if a value Falls in this region it is primarily an outlier and there should be a particular approach to cap it to a certain level so Z score greater than 3 or a P value whenever you are running a z test if it is less than 0.05 it falls in the critical Zone and that is an indicator that you should be not keeping those values when you are building a model so capping and flooring can be applied to both interquartile range and z-score the idea is that you keep the values higher values if the value is greater than Q3 values you cap them uh at that particular 75 percentile range and if it is a lower bound value let's say if it is a very low value it's lower than that 25th percentile value and you feel that does not make sense in the whole sense of the variable then you should be doing something known as flooring and you should cap it at the q1 value which is nothing but the 25th percentile similarly for capping with respect to the Z scores you should be considering the values you see at z z equal to 3 or Z equal to minus three because that is the range that still Falls in your confidence interval and that is fairly easy to understand that why it was capped under this range the other two commonly used approach and that might or might not fit in every use case is mean a median imputation and we are talking when we are talking about categorical Fields it's usually replacing the null values or the outlier values with the most highly frequent or the most low frequent uh category available in the data now when I'm talking on mean and median imputation you can obviously use this particular approach for treating the null values whereas for the outliers it might or might not make sense every time because sometimes an outlier is also related to other variables that are existing in the data it might be interdependent or correlated to any other field so before imputing based on mean or median you should always check whether the value makes sense with respect to other variables or not so that is one basic check that you should always keep in mind by doing these imputation exercises foreign I also wanted to share some real world examples that we I have also came across and now I have treated these uh Outlet here I am taking an example of a Google searched and for a specific keyword and I have extrapolated it across weeks here we can see there is a clear dip in the data and then again it after the dip there is a sudden spike in the data now these two could be an outline but based on my visual sense I know the trend cannot deviate so much so it can be classified as an outlet so when I do a normal distribution or a plot a histogram of this I can clearly see there are certain low values that are falling in the critical Zone and are not letting the distribution to look normal whereas if you see if you consider the distribution from here onwards it fairly looks like a normal distribution whereas these values should be treated in a way that it falls in a critical Zone I also went up with an approach of box plot as we were discussing it is basically checking whether how many values falls out of your interquartile range so there are certain High values and certain low values which I can clearly see are out of my confidence interval Zone and they I should take a call in capping or flooring them to make sense as I was previously mentioning outliers should also be looked from a business logic and here in this particular scenario this dip could be cause of a particular holiday that had a huge impact on the trend of that particular search keyword whereas it could also be an event that could cause and data outage that could be causing so there has to be always an explanation to be provided whenever you are capping or treating your outliers because it's important to understand why that particular thing has occurred in the data moving forward I would like to now touch base upon the next aspect of um data understanding or you can say this is the first step of feature engineering so feature Engineering in data science means that you are transforming your variable into a format that is best utilized or best fit when you are building a particular traditional model when you are saying data transformation meaning you're saying talking about data transformation and scaling approaches they help us in reducing the variability and the complexity of the data as well foreign I will be taking questions the last of the session so uh for anyone who's raising hand now talking about data transformation and benefits variables that process High bias and high variance and even after doing some kind of outline treatment you still feel there are certain values that are relating to skewness in your data and you need to amend those uh particular aspects in your data because a highly skewed data might not lead to a right outcome of the model as well so there has to be your sophisticated approach that you are a robust approach that you apply in transforming or scaling your data foreign algorithms whenever you are building your training data sets need these kind of approaches to be applied prior to them these are the algorithms that helps us in compressing our data in a particular range particularly talking about the continuous field because that is the range in which the model will behave the best else if the variance of the value or if taking a random example of a revenue field the value might range from 10 rupee to 10 lakhs so there is a high variability in the data and the model tends to deviate from the actual Behavior whenever this kind of variability exists so how to curve these or cater to these caveats that exist because of skewness or because of any variance that exist in the data we tend to resort to something that is called as scaling or transformation now there are a lot of techniques that are available but we need to understand when to apply and where to apply which kind of technique there are scaling techniques like Main X scalar which is also known as standardization there is Max absolute scalar which is uh in which we scale our values between minus one to one all these approaches are kind of used to normalize our distribution so the approach that we follow in any of these uh transformation technique is to create a normalized output of our data I will also be showing few examples of uh I will also be showing some examples where where I have taken some real world data example and have treated those and how do we see the transition from their skewness to a normal distribution so if I talk briefly talk about these transformation techniques it caters to a min max scalar which is scaling values between zero to one it's again uh compressing those a quantile transformation which is the most best fit transformation that is available out there so that also kind of treats your outlier and compresses the value between the quartile range log transformation is kind of uh or exponential transformation is some applied whenever you see a lot of skewness in the data and you cannot treat it with the basic techniques like we were previously discussing a normalizer is something that changes your mean to a zero value and your entire sum of your range would look uh correspond to a sum of absolute value one so these are some kind of transformation and how to visualize them and how they actually look in a data set I have given some example here now there was a happiness index data that is openly available on kaggle and I took the data set example from there now if you look at the there was in this data there was a score Fields available for generosity now the generosity field seems to be kind of skewed on the left side and when I apply a min max scalar or a standardized approach it kind of translates itself into a normal distribution now this kind of approach is important because now I have solved the problem of skewness in this and it is usable for in this log format there is a perception of corruption score available and clearly this is uh left skews after applying log transfer although it is still not ideally normally distributed but is fairly in a format that can you can use and help in building your model and taking example of rubber scalar it reduces the impact or compresses the values that are high and can be reduced to deter the values so these kind of values are helping or you can say are deviating us from a normal diffusion so robust scalar kind of suppresses those deviation values and gives you an output that is fairly looking like a normal distribution so these are kind of techniques that you should always keep on whenever you're building a regression or using any classification techniques because these models do have these assumption in place that if there is a standardization technique applied you will be getting a better outcome to whatever use case that you are solved I will take a pause and look at some question and answers that uh people have asked here so age can be uh both continuous data and nominal data it depends when if it is pinned in a categorical form let's say 10 to 20 20 to 50 and so on it will fall under the category of normal data but if it is given in a continuous format of an integer we can consider at us as in a continuous state go so Kramer's V and chi-square test of Industry Independence cramers we basically test you tells you an association it is basically a correlation approach that you apply for categorical varies whereas chi-square test basically tells you whether the both these things mean same or not two trend lines can be following the same Trend or two categories might be telling you the same thing but their values might be talking differently chi-square test also tells you that kind of Association that whether their means end up in telling the same story or not so whenever there could be a correlated field but to identify there could be two categories that might not be correlated to each other but could be still associated with each other in some other way that is why we take another approach of chi-square on top of Kramer's V that you can apply foreign talking about capping and flooring technique uh in a bit detail there we Whenever there is a high value or a spike in the data that doesn't resonate with the entire data fields you should be applying capping uh you should be applying something called capping it is basically masking those High values to a lower value and those lower value can be your choice it could be a 75th percentile value it could be a 95th percentile value but it should resonate with the entire flow of the data so whenever I'm talking about a normal Trend let's say an order purchase every week continuously we were seeing that the order was between a range of 8200 but suddenly in a certain day the orders were placed in excess of thousand so it could be a data tagging issue it could be a capturing issue so those in those kind of cases we have to cap and the two techniques that I was talking about are either ucab based on the z-score which is z minus 3 or Z3 which is just at the point or the words of the critical range and if you want to understand based on the interquartile range q1 or Q3 in short 25th percentile value of the continuous field or the 75th percentile huh so last question I will be talking about the LA the topic that we were previously covering which was transformation uh the how data transformation helps us in understanding or helps the model in understanding so you need to understand if there are large values the model will always behave or will be attributing a particular outcome based on the large values in the data so if they are scaled they are compressed in a particular range no value is discriminated for E from each other you don't want a field that is of our revenue and that is again compared against age or number of orders or similar low value continuous variable fields to be deviated and the entire Focus come into the revenue field because of the high volume that exists in the data So to avoid those kind of situations we transform data scale them it means 0 to 1 minus one to one that also enables Us in fitting the basic Assumption of most of the model that there should be a normal distribution in your particular numerical field that helps us in further um building the right model for it now even in categorical data there has to be some sort of feature of engineering to be applied these kind the two kind of basic steps that I would say are most commonly used are label encoding and one encoding what you can do is if you have a categorical fees that says salary high medium low is categorized you can encode them into one two three now for the model it will understand that one is for high medium is too low you can even rotate the encoding uh as per your choice you can put three as high medium S2 and 1 as Loop so that is encoding is asperature but the aim is to create it even to a continuous format the other way in which I feel is a better way of dealing with uh categorical Fields is creating binary Fields out of this so high for a high categorical row you will only have a binary flag one when you are corresponding or talking about that similarly for a salary medium low there will be medium associated in there and for a salary Loop in the on the lower row field it only will be flagged as one these two approaches will fairly help you in dealing with um how to put a categorical field in the data because most of the modeling techniques if you talk about svm logistical linear regression or any noise kind of approach you need to create them into a format that the model understands and the binary fields are the best kind of fields that you can generate that the model is easily will be able to understand and it is for you as an analyst is interpretable as well you can easily explain because of such value this kind of you is happening um now feature engineering as we talked about transformation techniques we talked about capping flooring we talked about how we can treat our Outlets they can both lead to increasing the dimensionality of their data and even reducing the damage that the end goal for us is reducing the dimensionality of the data so as long as the dimensionality of the data is controlled you will be able to train your model faster in even generate the output faster higher dimensional data takes a lot of time in training and even the output could be even scattered it could be deviated from the outcome that you are expecting to be so how can dimensionality of the data increase if you are let's say I was just taking an example of one eight one coding if let's say there was a categorical field of 25 categories and you do one hot encoding and you basically create binary feeds 25 binary Fields so that so that basically adding 25 new variables to the data similarly continuous variables when you derive them or convert them into a new format that also adds a new format when I'm talking about continuous variable you can be creating a lead lag variable out of it you could be creating bins out of those variables you could be bucketing them into particular thing and creating a new form of variable these all these things are to increase in the dimensionality when we are dealing with textual data we tend to create n grams of of those particular fields and that also adds an additional column basically they will cater to every word in that sentence of a field is capturing into a column is getting created into a column now that could be a binary column but overall the dimensionality of the data is increasing now what are the ways in which you can as a basic data science analyst how can you reduce those dimensionalities so we were we've been talking about multicolonarity that there could be a correlation test that you can do so when you are doing a correlation test you would already know that there are some combination of variables that are correlated to each other you should take a call whether which one to keep and which one not to keep that will further reduce some of the variables in the data now if you are not able to eliminate any of those you can further combine them together to make a particular fee so let's say there is a exam to say with an example there is a column that talks about spend and there is a column that talks about the revenue so rather than keeping both of them they might be spend or time consumed and rather than keeping both the correlated variables together you can create a ratio of time to spend in your data set and that might be enough to reduce the dimensionality rather than keeping two variables so always look for wherever you can combine certain variables together to make a new derived field because that always indicates to a better Enterprise interpretability of the model and you can explain it in a better format as well not keeping unwanted variables in the data so whenever you are when I'm talking about unwanted variables if you'll see a traditional database you will see Fields like customer ID time stamps um session ID product IDs these are the kind of fields that do not add any value to the data there are corresponding values that have some kind of value or in a continuous or a categorical format that you can utilize this information so it always makes sense that you should drop these variables and use the variables that are actually referenced to these uh particular keys now again similarly the way we use our statistics to compare whether two variables are associated to each other whenever you are solving a machine learning problem there will always be a dependent variable to check whether the independent variables all the other variables in the data set are related to it or not you can always check with an approach of T Test and Chi square and if they are not coming significant there isn't you can easily take a call on dropping these variables as well now talking to uh talking about some Advanced ways of reducing data dimensionality we will here speak about uh principle component analysis and factor analysis now um they did if you talk elaborate on these two particular topic it will again be another session but in nutshell what I wanted to tell you guys is the combi if you are taking a set of uh variables and they are equally important to you you can't take a call of dropping those variables and there is each info variable is providing some set of information but it is also enhancing the dimensionality or complexity of your data you can obviously go through an approach of something called the principal component analysis which is nothing but transform your variable into a set of two or three components which explains the entire variability of your data here the component one will explain let's say sixty percent of the variability of all the features that were there component two might further explain twenty percent so even if you are able to explain 80 to 85 percent variability of all the columns that are in your data you are fairly good in building a model whereas and you will be able to train it much faster than if you if we had a set of 10 columns or a 20 columns in your data similarly talking about factor analysis it is again a similar approach but here you do have some context of your variables in your data so you know that there are some variables that caters to only demographics so I can probably combine them to create one factor similarly uh there are certain variables that are related to let's say transactions of a particular product a transaction on a website and I can combine all those variables together to create one factor so that factor is nothing but it represents a linear output of a linear equation of these three variables so if you have let's say four or five transaction attributes in your data you can combine them together and uh because using the factor analysis model and it will give you a single output factor that kind of represents a linear output of those combinations so these are data dimensionality approaches that you should be using when you are not able to solve dimensionality problem uh you using the traditional approaches that we were talking in the previous slide and if there is a huge chunk of data available and you can't do to can't go on dropping variable by variable then I would highly recommend that you adopt techniques like PC and factor analysis which will further help you in understanding uh how the data dimensionality is getting reduced foreign the last aspect that we are going to cover here is the data wrangling or the major data angling uh approaches that we usually do in our day-to-day life so I've categorized them into pivoting which is basically spreading your data Outlaw or gathering them into a concise form it could be also aggregating into a form to make an uh to create an output of an analysis come forward so let's say we I'm taking a small example of a store detailed data and which in which you can see there is a date stamp there is a fiscal week available G of that particular it's a categorical field available uh you can also say this could be nominal now there is a source field that tells you whether it was an online purchase it was a store purchase or any other source now there is also a field for continuous Fields order place and revenue list so we do have a mix of all kind of uh variable attributes here we can clearly see uh orders uh place as continuous Fields source as a categorical Geo as a nominal and date timestamp also available now when I'm saying I want to spread my data it means I for a particular field I want to spread those categories as columns it could be existing at categories at one place it is similar to how we were doing 100 encoding but we are actually putting those real values in there so if you've seen this spread example what I've done here is the Geo column in which it has unique values of America's Asia Australia Europe I have spread that across the column so that kind of exercise helps you in understanding what are the cuts what is the different cuts that you can uh look at your data from you might also want to transform into a long format in which you just want only one continuous feed that could be used for any kind of exercise let's say you want to make it into a Time series format in which you only have a date and a value feed so in that format you what you have to do is you have to create your two continuous field into a categorical format in which you are basically doing into a gather you are gathering your data and what I've done is I've concise this order placed under when you call them into a category skill field and now we can clearly see there is a revenue field and Order Fields whereas their corresponding values are still in the continuous format talking about aggregation um you can summarize your data at any level you want um there are SQL operations and python operations available in which you can aggregate data at any level then the aggregation techniques that you can apply on on top of that could be related to summing those particular continuous Fleet taking a mean of those field taking a standard deviation or min max these are basic statistics that usually uh help us in doing most of the analysis there are other function customized function that people might kind of tend to use uh for for their aggregation technique so this is a gist of a data wrangling approaches that you might be using on day to day basis and it's good to be familiar with these kind of approaches because data stretching in the uh in a real world or real industry scope are kind of catering around these three aspects uh moving forward and I think we are only having five minutes and this is kind of the last uh touch Point uh available in our Uh current uh presentation so I wanted to touch base on the data day-to-day data that I have uh witnessed and I have worked on and you might also when you are starting your journey in data science you might also witness uh using these data sets so there could be a Time series data and mbz identifier of a Time series data there would be dates I feel and there would be a continuous field now this continuous field could be related to transaction could be related to web traffic could be related to anything but considering it's a continuous field along with the date time feel available you can fairly consider it as a use case that you can use to build uh forecasting technique um it could also be helping you in understanding what are the anomalies in this kind of data I normally detection is a good uh use case that is currently being used now in structure uh survey data is another example that I would like to uh touch base on this kind of data helps in um can be used in clustering and segmentation approaches here you can see there is no dependent variable available basically when I'm saying talking about dependent what is something that you could help in generating or as a predictable output of it these are basically using uh captured using Service uh people are rating each region based on what they feel on these particular aspects and it can help you in understanding which if our clustering use case can help you in understanding which set of countries are the most happiest they would be clustered together when you use that unsupervised approach and the most unhappiest would be also clustered in one kind of segment so that is one of the use case that whenever you see a data in which you there is not a clear dependent output available these are the kind of use case that you can obviously look into now in terms of a kind of an example of a semi-structured data I would say um these are um could be text review feedbacks that you might see on um even for a movie there are a lot of reviews there or if you go on a swiggy or a e-commerce platform there are uh kind of reviews there for each kind of product available so for these kind of textual data you need to First do a lot of cleaning but the kind of use cases that this solve is um could be in sentiment analysis could be in topic modeling you could be using these kind of use cases so there are a lot of use cases uh that you can always uh cater to from these kind of data sets I've also taken an example of a pharmaceutical data which basically has a dependent variable so this is one case where you can actually apply your classification techniques or regression techniques here we can see there are a lot of continuous variables available although there could be categorical available but we can further impute them into a binary class or anything that can help us in building a model but whenever you see an outcome field that is basically once you do uh that hits you whether you want to predict a positive or A negative outcome these are the use cases that directly hints you um towards that uh there could be a classification use case it could be where you are using your live based svm or logistic regression kind of model to further enhance or bring a predictable out so in nutshell these are like most commonly used data sets and you can obviously further explore a lot of data sets that are freely available on Google uh go on those go through those case studies and try to apply the things that we have learned in terms of data cleaning data understanding on top of that and whether that helps you in building a particular model um so yeah this was about the session and I think we are about time I'll quickly take few more questions uh that were came in the last and with that we will be for the ending the session yeah sure uh [Music] okay we'll talk some uh question answers from the uh attendees um so there are questions on some uh Nelson is our question on aegon values and again Vector introducing the dimensionality um so again values is basically the value that you generate out of so for whenever you run a PC or a factor analysis it it arranges your factor or components based on the eight values of the factor that is there so each variable each raw variable contributes to creating a component and again values is that associating uh score or associating Vector value that you can end up in relating to the raw value so let's say if a component was creating and it is if you have to associate with the raw variables you will always look at what are the aegon values given to those uh raw columns the to the columns that ended up creating that component and helps us in arranging and that also tells you that component could be highly dependent on one of the variables that was used in creating that so it is always good to understand how your component was created but it is fairly a next step to piece here that if you want to decode your PCA further uh that would be one thing that you should look into as per From eigenvalue perspective there is a question from himanshu on handling imbalance data uh yes that is one aspect that was not covered here uh just to briefly talk about how to handle imbalance data or category so usually in imbalance data might exist in categorical fields that there are certain categories that are having high values it could also be in a dependent attribute so there are certain kind of sampling techniques that are available what you basically do is for the categorical field that has a low frequency count you create duplicate or redundant rows of this and kind of add them append them into the bottom of your data sets that is one of the traditional ways when whereas a sophisticated approach of smart which is random over sampling is also available that you can use to create it basically creates duplicate values but also Alters the values of different columns to to the some extent so that it doesn't look a duplicate but you are also arranging them in your data data as to balance the categorical field to choose in terms of different type of encoding so I would honestly prefer you to go with one head on coding as when you are creating Whenever there is a high set of categories available in a column you would be encoding them from 1 to 50 let's say if there are 50 categories available now that again will further require you to scale them between a certain range So to avoid that it is always good to create them into a binary fields and it is easily interpretable also when you are building the model whenever you're doing a feature selection approach when you are doing a variable importance check also on your model these binary Fields come out better in terms of the work fields that were created as per label encoding um a good question on how do we understand whether to impute a mean or a median uh for handling null outliers so whenever you have a normal distribution normal distribution in your data that means your mean and median are fairly close to each other so in that case it you can absorb any technique but it is a good practice to always take median imputation that is the 50th percentile value which will be equidistant from both the standard deviation so your you are not even putting that one value you will not be deviating from the bell curve that gets created when you are checking the distribution so it's a good practice when you are not able to decide take the 50th percentile value um I think yeah we're good with the outstanding questions we had in in the session Now sort of yeah yeah uh thanks a lot Varun uh on behalf of analytics Vidya uh I would like to thank you for your time and uh for delivering such a wonderful session I'm sure our audience found it insightful and hopefully we can conduct more successions with you in the future thanks a lot