Python Tutorial: Exploratory data analysis

in this video I will show you how we can use exploratory data analysis to help identify data that need further investigation the most basic analysis we can do is count the unique values in our data we can use the info method to get the data type of each column here I will show you the frequency counts for the non numeric continent country and fertility columns and the numeric population column the performer frequency count we first select the column we want to perform a frequency count on if the column name does not contain any special characters spaces and is not a name of a Python function we can select the column directly by its name using dot notation it works the same way as subsetting using bracket notation once we have the column selected we can use the value counts method on the selected column I like to use the drop na equals false parameter since it will also count the number of missing values if there are any the continent column does not have a missing value so num will be reported value counts will print out the counts for each unique value of a column in descending order note that even though we counted a column of the object D type the results of value counts will be of the type int another way we can select columns is using the bracket notation here is the same code in output as before this time using bracket notation to select a column now we will count the number of observations for each country in our data since there are too many countries to show at once I'm using the head method to only return the top five counts in this example I'm chaining together methods I'm slicing and getting the value counts just like before we expect each country to have only one observation a Sweden has two this will require us to investigate this data point further the fertility column is the column we expect it to be numeric was stored as a string this is because we have a string name missing in the column this is why the fertility column has the wrong key type it also alerts us that we need to recode the missing string if your column has missing values it will also be counted provided you pass the drop na equals false parameter here you see we have 42 missing values in the column another type of EDA we can do is calculate summary statistics on numeric columns this can help spot outliers in our data there are many working definitions for outliers one definition is a value that is considerably higher or lower than the rest of the data you can consult the data can statistics course for more detailed definitions of outliers outliers are observations of interest we want to investigate further for data cleaning we can quickly calculate summary statistics on our data by using the describe method only the columns that have a numerical type will be returned describe returns a number of non missing values the mean standard deviation minimum 25th 50th and 75th percentiles of our data where the 50th percentile is the median and finally the maximum value of our data a quick scan down the population results show that the maximum population value is 2.3 billion people our data comes from 2012 no country had this population then now it's your turn to calculate descriptive statistics for exploratory data analysis to see what needs cleaning in your data