R Tutorial: Detecting anomalies in data

Key Takeaways

dealing with a normal hose datapoints also known as outliers is an important step in the data preparation phase if not properly handled outliers could skew your analysis and produce misleading conclusions in this video you will learn how to detect and treat outliers in your data to detect outliers in the distribution of a single numerical variable you can resort to two simple rules the three sigma rule for normally distributed data and the more general 1.5 IQR rule in a multivariate setting there is a plethora of methods to choose from we will highlight two of them the K nearest neighbors distance and the local outlier factor if your variable is normally distributed the three sigma rule can be applied in this case you can label any observations three standard deviations above or below the mean as an outlier given that its probability will be less than 0.1% another popular method is the 1.5 IQR rule this method can be applied regardless of your underlying data distribution the interquartile range or IQR is the difference between the third and the first quantiles of your variable any observation below p1 minus 1.5 IQR or above q3 plus 1.5 IQR is considered an outlier multivariate methods can be broadly categorized as either distance based or density based they both rely on the reasonable assumption that anomalous observations frequently live far from their neighbors distance based methods often use the average distance to their K nearest neighbors as an indicator of the extremeness of the observation density based methods operate by counting the number of neighboring points within a predefined reachability distance this plot shows some potential outliers along the fuel consumption and distance axis we can expect they will have a larger average distance to their neighbors compared to those points that appear clumped together we will use the get Canaan function from the fnn package to efficiently compute the distance matrix for the K nearest neighbors distance based methods fail to properly detect outliers in regions exhibiting different spatial densities these are called local outliers for example while the data point in the middle top of the figure could be regarded as a global outlier there might exist multiple local outliers in different regions the local outlier factor or LOF is an algorithm that measures the local deviation of a data point relative to its neighbors outliers are defined as data points with substantially lower density than their neighbors each of servation receives an lofs score that indicates whether it is deemed to be a regular data point an in liar or an outlier we will use the LOF function from the DBA scan package to compute the lofs score after detecting the outliers we can proceed in several ways first try to retain them in your data by choosing algorithms that are robust to them if this is not possible you could try applying data imputation methods such as linear or kanan imputation hopefully they will replace the outliers with less extreme observations another option is capping the outliers by replacing the low values with a 5th percentile of the variable distribution and the high values with a 95th percentile excluding the entire observations from your data analysis pipeline is also possible all this is often frowned upon undef initely not recommended for small datasets or those whose distribution is not Gaussian in any case use your domain knowledge to understand why these extremist relations are happening in your data set okay let