Machine Learning Engineer Full Course 2026 | Machine Learning Tutorial For Beginners | Simplilearn

Key Takeaways

Hi there, welcome to complete machine learning engineer full course. In today's AIdriven world, machines are no longer just following instructions. They are learning from data and improving over time. From recommendation systems to fraud detections to self-driving cars and intelligent chatbots, machine learning powers many of the technologies we use every day. Machine learning engineering goes beyond theory. It involves working with data, building models, optimization, performance, and also deploying solutions that can handle real world scale and complexity. In this complete machine learning engineer course, you will learn how to move from foundational concepts to build and deploying machine learning systems. We'll start with the basics of Python, mathematics, statistics, and then progress into core machine learning algorithms, model evaluation, and optimization techniques. As you advance, you will also explore important areas such as feature engineering, deep learning, model deployment, and envelopes practices. You'll also gain hands-on experience with industry tools, and libraries used to build and manage machine learning workflows. By the end of this course, you'll have a strong understanding of how machine learning models are built, evaluated, and deployed in production environments. Having said that, let's take a look at today's agenda for this course. We'll start off with module one which is introduction to machine learning and AI. Module two is Python programming for machine learning. Module three is mathematics and statistics for ML. Module four is data collection and data prep-processing. Module five is exploratory data analysis. Module six is feature engineering and feature selection. Module seven is supervised learning algorithms. Module 8 is unsupervised learning algorithms. Module 9 is model evaluation and performance metrics. Module 10 is introduction to deep learning. Module 11 is model deployment basics. Module 12 is envelopes and model life cycle management. Module 13 is real world machine learning projects. Module 14 is interview question and career guidance. Hope I made myself clear with this agenda. That said, if these are the type of videos you'd like to watch, then hit that subscribe button with the bell icon to get notified whenever we post. Also, just so that you know, if you want to upskill yourself, master generative AI and land your dream job or even grow in your career, then you must explore SimplyLearn's cohort of various generative AI training and professional certificate programs. Simply offers a variety of master certification and post-graduate programs in collaboration with some of the world's leading universities. Through our courses, you will gain knowledge along with work ready expertise in skills like Python, Agentic AI, AI automation systems, LLMS, and over a dozen others. And that's not all. You'll also get the opportunity to work on multiple projects led by industry experts working in top tier service- based and product companies. After completing these courses, thousands of learners have transitioned into AI and machine learning role as a fresher or moved on to a higher paying job and profile. If you're passionate about making your career in this field, then make sure to check out the link in the pin comments and in the description box to find an AI and machine learning program that fits your experience and areas of interest. So let's get started with our machine learning engineer full course with a small quiz. Which type of learning uses label data? Is it unsupervised learning, supervised learning, reinforcement learning, deep learning? Please let us know your answers in the comment section below. Now over to our training experts. We know humans learn from their past experiences and machines follow instructions given by humans. But what if humans can train the machines to learn from their past data and do what humans can do and much faster? Well, that's called machine learning. But it's a lot more than just learning. It's also about understanding and reasoning. So today we will learn about the basics of machine learning. So that's Paul. He loves listening to new songs. He either likes them or dislikes them. Paul decides this on the basis of the song's tempo, genre, intensity, and the gender of voice. For simplicity, let's just use tempo and intensity for now. So, here tempo is on the x-axis, ranging from relaxed to fast, whereas intensity is on the y-axis, ranging from light to soaring. We see that Paul likes the song with fast tempo and soaring intensity while he dislikes the song with relaxed tempo and light intensity. So now we know Paul's choices. Let's say Paul listens to a new song. Let's name it as song A. Song A has fast tempo and a soaring intensity. So it lies somewhere here. Looking at the data, can you guess whether Paul will like the song or not? Correct. So Paul likes this song. By looking at Paul's past choices, we were able to classify the unknown song very easily, right? Let's say now Paul listens to a new song. Let's label it as song B. So song B lies somewhere here with medium tempo and medium intensity. Neither relaxed nor fast, neither light nor soaring. Now, can you guess whether Paul likes it or not? Not able to guess whether Paul will like it or dislike it. Are the choices unclear? Correct. We could easily classify song A. But when the choice became complicated as in the case of song B. Yes. And that's where machine learning comes in. Let's see how. In the same example for song B, if we draw a circle around the song B, we see that there are four votes for like whereas one vote for dislike. If we go for the majority votes, we can say that Paul will definitely like the song. That's all. This was a basic machine learning algorithm also. It's called K nearest neighbors. So this is just a small example in one of the many machine learning algorithms quite easy right believe me it is but what happens when the choices become complicated as in the case of song B that's when machine learning comes in it learns the data builds the prediction model and when the new data point comes in it can easily predict for it more the data better the model higher will be the accuracy there are many ways in which the machine learns it could be either supervised learning unsupervised learning or reinforcement learning. Let's first quickly understand supervised learning. Suppose your friend gives you 1 million coins of three different currencies. Say 1 rupee, 1 and 1 dirham. Each coin has different weights. For example, a coin of 1 rupee weighs 3 g. 1 euro weighs 7 g and 1 dirham weighs 4 g. Your model will predict the currency of the coin. Here your weight becomes the feature of coins while currency becomes their label. When you feed this data to the machine learning model, it learns which feature is associated with which label. For example, it will learn that if a coin is of 3 g, it will be a 1 rupee coin. Let's give a new coin to the machine. On the basis of the weight of the new coin, your model will predict the currency. Hence, supervised learning uses labeled data to train the model. Here, the machine knew the features of the object and also the labels associated with those features. On this note, let's move to unsupervised learning and see the difference. Suppose you have cricket data set of various players with their respective scores and the wickets taken. When we feed this data set to the machine, the machine identifies the pattern of player performance. So, it plots this data with the respective wickets on the x-axis while runs on the y-axis. While looking at the data, you'll clearly see that there are two clusters. The one cluster are the players who scored high runs and took less wickets while the other cluster is of the players who scored less runs but took many wickets. So here we interpret these two clusters as batsmen and bowlers. The important point to note here is that there were no labels of batsmen and bowlers. Hence the learning with unlabeled data is unsupervised learning. So we saw supervised learning where the data was labeled and the unsupervised learning where the data was unlabeled. And then there is reinforcement learning which is a reward-based learning or we can say that it works on the principle of feedback. Here let's say you provide the system with an image of a dog and ask it to identify it. The system identifies it as a cat. So you give a negative feedback to the machine saying that it's a dog's image. The machine will learn from the feedback and finally if it comes across any other image of a dog, it'll be able to classify it correctly. That is reinforcement learning. To generalize machine learning model, let's see a flowchart. Input is given to a machine learning model which then gives the output according to the algorithm applied. If it's right, we take the output as our final result. Else we provide feedback to the training model and ask it to predict until it learns. I hope you've understood supervised and unsupervised learning. So let's have a quick quiz. You have to determine whether the given scenarios uses supervised or unsupervised learning. Simple, right? Scenario one. Facebook recognizes your friend in a picture from an album of tagged photographs. Scenario two, Netflix recommends new movies based on someone's past movie choices. Scenario three, analyzing bank data for suspicious transactions and flagging the fraud transactions. Think wisely and comment below your answers. Moving on, don't you sometimes wonder how is machine learning possible in today's era? Well, that's because today we have humongous data available. Everybody's online either making a transaction or just surfing the internet and that's generating a huge amount of data every minute and that data my friend is the key to analysis. Also, the memory handling capabilities of computers have largely increased which helps them to process such huge amount of data at hand without any delay. And yes, computers now have great computational powers. So there are a lot of applications of machine learning out there. To name a few, machine learning is used in healthcare where diagnostics are predicted for doctor's review. The sentiment analysis that the tech giants are doing on social media is another interesting application of machine learning. Fraud detection in the finance sector and also to predict customer churn in the e-commerce sector. While booking a cab, you must have encountered search pricing often where it says the fair of your trip has been updated. Continue booking. Yes, please. I'm getting late for office. Well, that's an interesting machine learning model which is used by global taxi giant Uber and others where they have differential pricing in real time based on demand, the number of cars available, bad weather, rush hour, etc. So they use the search pricing model to ensure that those who need a cab can get one. Also, it uses predictive modeling to predict where the demand will be high with a goal that drivers can take care of the demand and search pricing can be minimized. Great. Hey Siri, can you remind me to book a cab at 6 p.m. today? >> Okay, I'll remind you. >> Thanks. >> No problem. >> Comment below some interesting everyday examples around you where machines are learning and doing amazing jobs. So, that's all for machine learning basics today from my side. Keep watching this space for more interesting videos. Until then, happy learning. >> Artificial intelligence, machine learning, and deep learning represent the evolution of computer science towards creating intelligent systems. AI is the broader concept striving to build machines capable of humanlike intelligence. ML is a subset of AI emphasizing algorithms that learn from data to make predictions or decisions. DL in turn is a specialized branch of ML that employs deep neural networks to model complex patterns. Imagine an AI powered voice assistant like Apple Siri. It utilizes ML to understand and respond to user queries, learning from interactions over time. Deep learning comes into play when Siri recognizes speech patterns or interprets natural language using neural networks to process intricate features. The better it becomes at understanding diverse accents or refining responses. Exemplifying the continuous learning inherent in these technologies. AI seeks to emulate human intelligence. ML harness data for learning and DL employs deep neural networks for intricate task. The integration of these technologies manifest in everyday applications transforming how we interact with and benefit from intelligent systems. This technology enables voice interaction allowing the device to play music, set alarms, present audio books and provide upto-date information on topics like news, weather, sports and traffic reports etc. Let's move forward and see what is machine learning. Machine learning is a subset of artificial intelligence that focuses on developing algorithms and models capable of learning and making predictions or decisions without being explicitly programmed. ML systems leverage data to recognize patterns, adapt and improve their performance over time. There are several types of machine learning. Number one, supervised learning. The algorithm is trained on a label data set where each input is associated with a corresponding output. Number two comes as unsupervised learning. Unsupervised learning deals with unlabelled data to find inherent patterns or structures within the information. And then comes the reinforcement learning. This type involves training agents to make sequences of decisions by interacting with an environment. And then comes semi-supervised learning. Semi-supervised learning combines supervised and unsupervised learning elements typically using a small amount of labelled data and a larger pool of unlabelled data. Let us move forward and see what deep learning is. Deep learning, a branch of machine learning, focuses on algorithms inspired by the human brain structure and functionality. It excels in processing vast amounts of both structured and unstructured data. At the heart of deep learning are artificial neural networks, empowering machines to make decisions. The key distinction between deep learning and machine learning lies in data presentation. Machine learning algorithms typically demand structured data while deep learning networks operate through multiple layers of artificial neural networks allowing them to handle diverse data formats. So let's start with the difference between artificial intelligence, machine learning and deep learning. And this we'll show in a table form. So starting with the definition. So definition of artificial intelligence. So broad field of machine learning or creating machines with intelligent behavior is artificial intelligence. And when we talk about machine learning, it's the subset of AI focusing on algorithms learning from data. And then comes the deep learning that is specialized subset of ML using deep neural networks. And now we'll see the difference with the learning approach between all these three. So in learning approach artificial intelligence can include rule-based systems, expert system and more. And in machine learning it learns from data patterns without explicit programming. And then comes the deep learning where it learns hierarchical representation using neural networks. And if we talk about scope, it encompasses various techniques beyond learning from data. And in machine learning, it primarily focus on learning patterns from data. And then the deep learning, it specifically utilizes deep neural networks for complex task. And now we'll move to the next difference and we'll start with an example. So in artificial intelligence, the example is autonomous vehicles, chatboards or expert systems. And for machine learning, it's spam filters, recommendation systems, image recognition. And in deep learning, it is image and speech recognition, natural language processing. And now we'll see the difference for the data requirements. So it depends on the specific application and problem solving approach. And in machine learning, it requires labeled or unlabelled data or training. And for the deep learning, it relies on large amounts of labelled data for training deep networks. And now for the complexity, artificial intelligence addresses a wide range of task including those beyond ML. And in machine learning, it deals with moderate to complex task depending on algorithms. And for the deep learning, it is well suited for intricate task often requiring substantial computational resources. And now see the flexibility. So for the artificial intelligence, it can be rule-based, evolving and adaptive. And for the machine learning, the flexibility adapts to patterns and the changes in data. And for the deep learning, it adapts to hierarchical representations and diverse data types. And then comes the training process. So in artificial intelligence, training process varies based on specific AI techniques used. And in machine learning, training involves feeding data and adjusting model parameters. And in deep learning, training involves optimizing neural weights and structures. And now we'll talk about the applications between all these three terms that is A IML and deep learning. So for artificial intelligence the applications are robotics, natural language processing, game playing. And for machine learning it's predictive analytics, fraud detection and healthcare diagnostic and for the deep learning that is image recognition, speech synthesis and language translation. We have Rahul who will take you through the various applications of machine learning and how you can be a machine learning engineer. >> Machine learning has improved our lives in a number of wonderful ways. Today let's talk about some of these. I'm Rahul from SimplyLearn and these are the top 10 applications of machine learning. First let's talk about virtual personal assistants. Google Assistant, Alexa, Cortana and Siri. Now we've all used one of these at least at some point in our lives. Now these help improve our lives in a great number of ways. For example, you could tell them to call someone. You could tell them to play some music. You could tell them to even schedule an appointment. So, how do these things actually work? First, they record whatever you're saying, send it over to a server, which is usually in a cloud, decode it with the help of machine learning and neural networks, and then provide you with an output. So, if you ever notice that these systems don't work very well without the internet, that's because the server couldn't be contacted. Next, let's talk about traffic predictions. Now, say I wanted to travel from Buckingham Palace to Lots Cricket Ground. The first thing I would probably do is to get on Google Maps. So, search it and let's put it here. So, here we have the path you should take to get to Lord Cricket Ground. Now here the map is a combination of red, yellow and blue. Now the blue regions signify a clear road that is you won't encounter traffic there. Yellow indicate that they're slightly congested and red means they're heavily congested. So let's look at the map a different version of the same map. And here as I told you before red means heavily congested, yellow means slowmoving and blue means clear. So how exactly is Google able to tell you that the traffic is clear, slowmoving or heavily congested? So this is with the help of machine learning and with the help of two important measures. First is the average time that's taken on specific days at specific times on that route. The second one is the real-time location data of vehicles from Google Maps and with the help of sensors. Some of the other popular map services are Bing Maps, Maps.me. And here we go. Next up, we have social media personalization. So say I want to buy a drone and I'm on Amazon and I want to buy a DJI Mavic Pro. The thing is it's close to one lap, so I don't want to buy it right now. But the next time I'm on Facebook, I'll see an advertisement for the product. Next time I'm on YouTube, I'll see an advertisement. Even on Instagram, I'll see an advertisement. So here with the help of machine learning, Google has understood that I'm interested in this particular product. Hence, it's targeting me with these advertisements. This is also with the help of machine learning. Let's talk about email spam filtering. Now, this is a spam that's in my inbox. Now, how does Gmail know what's spam and what's not spam? So, Gmail has an entire collection of emails which have already been labeled as spam or not spam. So after analyzing this data, Gmail is able to find some characteristics like the word lottery or winner. From then on, any new email that comes to your inbox goes through a few spam filters to decide whether it's spam or not. Now, some of the popular spam filters that Gmail uses is content filters, header filters, general blacklist filters, and so on. Next, we have online fraud detection. Now, there are several ways that online fraud can take place. For example, there's identity theft where they steal your identity. fake accounts where these accounts only last for how long the transaction takes place and stop existing after that and man-in-the-middle attacks where they steal your money while the transaction is taking place. The feed forward neural network helps determine whether a transaction is genuine or fraudulent. So what happens with feed forward neural networks are that the outputs are converted into hash values and these values become the inputs for the next round. So for every real transaction that takes place there's a specific pattern. A fraudulent transaction would stand out because of the significant changes that it would cause with the hash values. Stock market trading machine learning is used extensively when it comes to stock market trading. Now you have stock market indices like Nikai. They use long short-term memory neural networks. Now these are used to classify, process and predict data when there are time lags of unknown size and duration. Now this is used to predict stock market trends. Assistive medical technology. Now medical technology has been innovated with the help of machine learning. Diagnosing diseases has been easier from which we can create 3D models that can predict where exactly there are lesions in the brain. It works just as well for brain tumors and ice schemic stroke lesions. They can also be used in fetal imaging and cardiac analysis. Now, some of the medical fields that machine learning will help assist in is disease identification, personalized treatment, drug discovery, clinical research, and radiology. And finally, we have automatic translation. Now, say you're in a foreign country and you see billboards and signs that you don't understand. That's where automatic translation comes of help. Now how does automatic translation actually work? The technology behind it is the same as the sequence to sequence learning which is the same thing that's used with chatbots. Here the image recognition happens using convolutional neural networks and the text is identified using optical character recognition. Furthermore, the sequence to sequence algorithm is also used to translate the text from one language to the other. And today I'm going to tell you how you can become a machine learning engineer. But before we begin, let me tell you what a machine learning engineer actually does. A machine learning engineer creates and maintains machine learning solutions to solve business problems. They constantly tweak and optimize the solutions for maximum performance and scalability. They solve business problems like reducing customer churn, running targeted marketing campaigns, and improving product experience. They also help with predicting whether a particular hypothesis will be profitable in the future. They contribute to cutting edge research in AI and machine learning. Now before you can start off on your journey to becoming a machine learning engineer, there's a certain number of steps that you need to follow. On this learning path, your first step is to improve your math skills. Mathematics plays a very important role in helping you understand how machine learning and its algorithms work. Among the many concepts that you need to understand, three of the most important ones are probability and statistics, linear algebra, and calculus. Now let's have a detailed look at probability and statistics. Firstly, you have the base theorem. This is used in the nave base algorithm to help categorize your data. Then we have probability distribution. This helps you determine how frequently a particular event will take place. For example, you can determine what your premium for your insurance will be based on the probability distribution of expenditure pertaining to insurance claims. You must also learn how sampling and hypothesis testing works. Now let's look at linear algebra. Now linear algebra has two main concepts, matrices and vectors. They're both used widely in machine learning. Now, matrices are used for image recognition where the entire image that you're using is already in the form of a matrix. You need to be able to work with matrices, perform simple operations like addition, subtraction, multiplication, inverse, transpose and so on. Now, the recommended systems that you see in applications like Netflix or Amazon actually work on vectors. This vector is the customer behavior vectors and they use distance measures. Now, let's have a closer look at calculus. You have differential calculus and integral calculus. Now these help in determining the probability of events. For example, in finding the posterior probability in a navebased model. In your next step to becoming a machine learning engineer, you need to develop good programming skills. And let me tell you, there are a huge number of options from which you can choose. There's Python, there's C, there's C, there's Java and so much more. Now, here's a graph of the job postings from 2014 to 2017. And you can see that there are two languages that dominate since 2015. Python and R. Now these are one of the most wanted languages when it comes to machine learning engineers. These are closely followed by JavaScript and C. Hence, we would like to recommend that you learn Python and R as they're the best options when it comes to coding in machine learning algorithms. So here are a few things that you need to know about Python and R. Python is an object-oriented language which means their main emphasis is on the object. R is a functional language which means their emphasis is on creating and manipulating functions. Python relies on its many packages and R is slightly faster than Python because it has inbuilt packages. Now, Python is generic and is suitable if you need to integrate it with any other software. R works a little more closer with statistical analysis. Your next step is to get yourself some data engineering skills. Now, these skills are important as they help you analyze and process your data as soon as you get it. Now there are three major steps when it comes to data engineering. Firstly, there's data prep-processing. Now these refer to all the steps that you need to perform before data can be processed by the machine learning algorithm. You have cleaning, parsing, correcting and consolidating the data. Then you have ETL or extract, transform and load. You need to know how data can be extracted from the internet or a local server. You need to know how to transform the data. For example, not all formats of data would be accepted by the program. So you need to convert the data into a format that is accepted by the program. Then you need to know how the data is loaded into your program. The final step is to have knowledge about database management softwares or DBMS. You need to be well versed with MySQL, the Oracle database and NoSQL. Your next step is to learn machine learning algorithms. Now among all the machine learning algorithms that you can see on screen, you can divide them into two different categories. ones that fall under supervised machine learning and the others that fall under unsupervised machine learning. Even after such a division, you can further subdivide them into classification and regression algorithms. Now, all algorithms except linear regression fall under the category of classification which is used to determine whether a particular data falls into a particular category. On the other hand, you have linear regression that falls under regression algorithms. This is used to predict a particular value. Then you have K means clustering and hierarchal clustering. Now these fall in the category of clustering. This is used to group data into clusters based on certain similar attributes. Then you have the a priori algorithm that falls under the concept of association. Association is used to determine patterns of association among variables in large data sets. So now that you know about these algorithms, let me tell you where you can learn about them. Let's take a look at our simply learn channel and let's go to playlists. And on this we have a dedicated set of playlists that talk about machine learning. Here you have videos on machine learning. How machine learning is different from deep learning and artificial intelligence. Machine learning with Python, K means clustering, decision trees and so on. If you want to learn more, you could also go through some deep learning algorithms like convolutional neural networks, recurrent neural networks, long short-term memory networks and so on. In fact, we have a detailed playlist that talks about the concepts of deep learning. Here you can find videos on what is deep learning, TensorFlow, what is a neural network, convolutional neural networks, and recurrent neural networks. Now, after you master these algorithms, you need to learn how you can select the right algorithm for your problem. Then you need to create a good model with one or more algorithms. after which you need to keep tweaking and optimizing the model so that you can get the maximum accuracy. For further reading, you can go through GitHub where there's more than 21,000 repositories under machine learning. You can also recreate published research papers adding to your experience of working with machine learning. And now for your last step, learning machine learning frameworks. Machine learning frameworks help make the lives of developers as well as users a whole lot easier. They help remove the complex part of machine learning and make it available for everyone who wants to use it be it developers or other users. Now let's look at the widely used machine learning frameworks. There's TensorFlow, Theiano, Torch, Scikitlearn and so on. Now let's look at some of them in detail. First let's look at TensorFlow. TensorFlow is the most widely used machine learning framework. It's used for machine learning as well as deep learning. Now it's an open-source software library which performs numerical computations which is done with the help of dataf flow graphs. Google translate is one of the most popular use cases of TensorFlow. Now let's look at Theano. Theo helps you define, optimize and evaluate mathematical expressions. It was developed in the University of Montreal. Lasani blocks and keros are its most popular libraries. Now let's look at Spark ML library. Now this is Apache Spark's machine learning component. It also provides libraries for machine learning which are built on top of RDDs or resilient distributed data sets. It's very good at providing iterative computation and provides very high algorithmic performance. Now let's look at scikitlearn. Scikitlearn is able to provide a huge range of supervised as well as unsupervised algorithms for machine learning. It is built on existing libraries like numpy, scypi and mattplot lib. Scikitlearn actually started off as a Google Summer of Code project and now has 23,000 GitHub commits. And that's it. Congratulations, you're now a machine learning engineer. Now, let's look at the job opportunities in machine learning. Now, if you have a look at this graph, it kind of speaks for itself. Before 2015, machine learning was much less popular than big data and cloud computing. But all of that suddenly changed. And right now, a machine learning engineer earns around $114,000 perom. And this is a clear indication that organizations are ready to invest heavily in people who are skilled in the concepts of machine learning. You can also make the learning process easier by using simply learns machine learning certification. Simply learns machine learning certification course provides 36 hours of instructorled training. Provides 25 plus hands-on exercises. gives you practical applications of 15 plus machine learning algorithms and helps you master the concepts of supervised and unsupervised learning. It also introduces you to artificial intelligence, tells you the techniques of machine learning, data processing, regression, classification and so much more. So if you want to take your first step to getting certified and getting ahead >> offline now machine learning systems are integrated into real world operations and are capable of making decisions that impact the business immediately. Here's an example to make it clearer. In the past, an e-commerce website might use machine learning to predict what products a customer might want based on their past purchases. Now the systems can constantly learn from new customer data continuously refining those predictions in real time as customers preferences are changing. The world of machine learning has evolved from theory to practice and this has created a huge demand for machine learning engineers who can build scalable systems and make them work in real world environments. The impact of machine learning is now directly tied to business outcomes and machine learning engineers are at the center of that transformation. You might be thinking, okay, I get it. Machine learning is impactful, but what exactly does a machine learning engineer do compared to other roles in tech? That's a great question. In the world of machine learning, you'll hear about a few key roles such as data scientist, machine learning, and AI engineer. Let's break them down so you know exactly where you fit in. Data scientists are like the detectives of data. They spend their time analyzing large data sets, finding trends, and trying to extract meaningful insights. They build models but their main focus is usually on data exploration and experimenting with various algorithms. They don't typically focus on deploying those models into production environments. Machine learning engineers on the other hand these are architects. They take the models built by data scientists and build scalable deployable systems. They work on creating solutions that will not only work in the short term but can also scale to handle real world data in massive volumes. The machine learning engineer is responsible for ensuring that machine learning systems are integrated into businesses that can work seamlessly with existing technologies. AI engineers focus more on the application side of things. They build AI powered products like chatbots, voice assistants, and real-time systems. While their work often overlaps with ML engineers, they are typically more focused on the userfacing product and how machine learning fits into it. As an ML engineer, your primary focus is to take models and turn them into actionable solutions that are deployed in real world systems. We shall now move on to why 2026 is the right time to enter machine learning. Now that you know the role of an ML engineer, let's talk about why 2026 is the perfect time for you to jump into the field. You've probably heard that machine learning is a hot topic, but what does that mean for you as someone starting out in this field? So, I'll help you break that down for you. First, the demand for ML engineers has skyrocketed. The world is full of problems that need solving, and machine learning has proven to be one of the most effective tools to solve them. From predicting customer preferences to automating critical business functions, machine learning is changing how businesses operate. Secondly, the tools used to build machine learning systems are more accessible than ever before. In the past, machine learning was viewed as something experimental, something that required a lot of effort just to set up. But today machine learning platforms and frameworks such as TensorFlow, PyTorch and Scikitlearn have matured significantly. These tools make it easier to build and deploy models that can scale to handle real world data. We shall now move on to why is this the right time for you to get started out as a machine learning engineer. So how do you get started? The first step is to build a strong foundation. You might be excited to start building models and diving into algorithms. But before that you need to understand the core concepts that drive all the machine learning systems. These include mathematics, programming and data handling. You don't need to be an expert in all of these areas, but you do need to understand the basics. Think of these as building blocks of everything that you will need to learn machine learning. Let's start with mathematics. You don't need to be a math genius, but you do need to understand the basics. There are three main areas of math that will help you get started out as an ML engineer, and those are linear algebra. This is the study of vectors and matrices which are used to manipulate and process data in machine learning models. If you have heard terms like feature vectors or matrix operations, that's linear algebra at play. This is the core of how machine learning algorithms operate. This is the math behind optimizing models. You'll use calculus to adjust the parameters of machine learning models and minimize the error between predicted and the actual values. Specifically, derivatives are used to find the best way to fit a model into the data. Statistics. Machine learning is all about working with uncertaintity, and statistics will help you make sense of it. Whether you're dealing with probability distributions or hypothesis testing, statistics can help you understand patterns in data and make decisions with uncertain information. These mathematical concepts will help you build a more accurate model and optimize it effectively. Let's move on to programming stack. If you're new to programming, don't worry. Python is the language that you will want to learn. It's simple to get started with and has a huge ecosystem of libraries especially designed for machine learning. The key libraries that you will need to master include numpy. This library is used to handle large arrays of matrices and data which is the backbone of most machine learning algorithms. If you plan on working with large data sets, you'll be using numpy a lot. Pandas. This library is great for data manipulation and analysis. You'll use pandas to clean, organize, and transform data, making it ready for machine learning models. Scikitlearn. This library provides simple, easy to use tools for building machine learning models. It covers everything from data prep-processing models like regression and classification. Once you're comfortable with these tools, you'll be able to start building machine learning models and working with real world data. Next up is SQL, which stands for structured query language. As an ML engineer, you will be working on lots of data, and SQL will help you query databases to retrieve information that you need. You'll use SQL to extract data, filter it, and join tables together, making sure that you have the right data to your models. Once you have your data, the next step is data wrangling. Data wrangling is a huge part of your job, and if you master it, it will save you a lot of time and frustration while building models. We shall now move on to types of machine learning. Let's talk about the different types of machine learning. There are three main categories which are supervised learning, unsupervised learning and reinforcement learning. Speaking of supervised learning, this is when you have label data. You train your model on data where the answers are already known. The goal is for the model to learn the relationship between inputs and outputs so it can predict the future outcomes. Unsupervised learning. In this case, the model works with unlabelled data and it tries to find hidden patterns and groupings in the data. This is useful for tasks like clustering or anomaly detection. Reinforcement learning. This type of learning involves training an agent to make decisions by interacting with its environment and receiving rewards or penalties based on its actions. This is often used in game AI or robotics. We shall now move on to feature engineering and evaluation. After cleaning your data, the next step is feature engineering. This is the process of transforming raw data into meaningful features that help your model make better predictions. Once you've engineered your features, it's time to evaluate the model. You'll use metrics like accuracy, precision, and recall to see how well your model is performing. Proper evaluation ensures that your model is ready for real world applications and that it can handle data in production. We shall now move on to the six-month learning plan in order to become an ML engineer in 2026. So, how do you actually get started? Here's your six-month learning plan to guide your journey. Firstly, focus on learning Python, mathematics, and SQL. Dive into machine learning algorithms and hands-on projects. Then, learn deep learning and work with frameworks like TensorFlow or PyTorch. By the sixth month, you can work on a capstone project that covers the full pipeline from data collection to deployment. We shall now speak about the ML engineer tool stack. So let's talk about the tools that you will need as a machine learning engineer. You must be wondering what tools should I be learning to become a successful ML engineer in 2026. So let's simplify this. Git and GitHub are the first things on your list. At first glance, version control might seem like something that only coders need to worry about. However, it can be really crucial. Why? Because version control allows you to track every change that you make to your code, collaborate with teams, and roll back changes if something goes wrong. Imagine you're working on a huge project and you mess something up. Without version control, you could lose hours of work. But with Git and GitHub, you could go back and fix to any point and time. You might be thinking, okay, I get that Git is useful, but what about the tools that can actually help me build and deploy machine learning models? Now, that's a great question. So let's talk about the cloud platforms like AWS, Google Cloud, and Azure. In the past, machine learning models were often built and tested upon local machines, but we quickly realized that it wasn't scalable. These cloud platforms give you the ability to handle large data sets, run models on highowered servers, and scale them as your projects grow. Instead of relying on your personal laptop to do all of the heavy lifting, you can leverage the cloud to train models much faster and handle massive data sets without worrying about memory limitations. We shall now move on to the next topic which is on MLOps life cycle. You have now got all of your tools in place. But how do you go from all of these tools coming together in the MLOps life cycle? So you must be wondering what does all of the MLOps life cycle tools even look like and how do I manage the process from starting to finish? Well, here's the thing. MLOps is like a welloiled machine. It involves a series of stages that ensure that your models remain reliable, scalable, and adaptable to new data over time. Think of it like a car assembly line. First, you train the car and then you deploy it into the real world. After that, you need to monitor it and see if it runs smoothly. If it breaks down, you will have to retrain it. Let's break it down into key steps. Training your model. This is where you create the model using your training data. This is a very experimental phase where you try different algorithms, tweak parameters, and optimize the model for better performance. Deploying it to production. Now that your model is ready, it's time to put it into action. This means making the model available to users or clients. Whether it's in a mobile app or a web service, deployment is where the magic happens. Monitoring performance. Once your model is live, you can't just forget about it. It's like checking the car's tire pressure after it's been on the road for a while. You need to continuously track how well your model is doing. If it starts to slip or underperform, it's time for tweaks and adjustments. Lastly, we have retraining. Over time, your model may need to be retrained with new data to stay accurate. This is especially true in industries where the environment is constantly changing, like e-commerce or finance. Restraining ensures that the model stays relevant and continues to provide value. You may be wondering that this sounds like a lot of work and that's true. That's where MLOps tools like MLOR help you streamline this process. By automating and managing this life cycle, you can spend less time dealing with the back end and more time focusing on building innovative solutions. We shall now move on to experiment tracking. As you dive deeper into machine learning, you'll quickly realize how important it is to track your experiments. At first, this might seem a little overwhelming. You might be thinking, I can't just run a model and hope for the best, right? But trust me, tracking your experiments is one of the best habits that you can develop early on. Think of it like logging your workout progress. Even if you don't track the results, how will you know whether you're improving or not? The same goes for machine learning. By using experiment tracking tools flow and weights and biases, you can keep a detailed record of every model and every hyperparameter and every evaluation metric. Now imagine you're building a recommendation system for an online store. You try different adjustments, algorithms, and get different results. With experiment tracking, you can easily compare which configurations work best. You can easily compare which configurations work best and learn from the past mistakes. You'll always know which experiment gives you the best results and which needs tweaking. Tracking experiments also helps in collaboration. If you're working with a team, being able to see everyone's experiments in one place makes it easier to understand their approach and build on each other's work. We shall now move on to projects and portfolio. Now that you've got your tools and workflow in place, it's time to focus on building a strong portfolio. You might be thinking, how do I make my portfolio stand out to potential employees? That's a great question. The answer is simple. Real world projects. Think about it. Employers want to see what you can do in practice. They don't just want to see theoretical knowledge. They want to see that you can solve problems and build working systems that can make a difference. So for that reason, we have mini project examples. At this point, you're probably itching to start building something yourself. Well, hands-on projects are the best way to solidify what you've already learned. For example, you could create a customer churn prediction model or a fraud detection system. These are practical real world projects that demonstrate your ability to build solutions from start to finish. Make sure you document your projects clearly on GitHub and always include a detailed explanation of your approach, challenges, and results. Remember, the goal is to show that you understand the problem and build a working solution. We'll now move on to Kaggle and open source. As you continue to build your portfolio, I highly recommend diving into Kaggle competitions and contributing to open source projects. Kaggle is an amazing platform where you can work on real world data sets and solve problems that companies and research institutions are facing. Not only will you improve your skills, but you'll also have the opportunity to see how top data scientists and machine learning engineers approach similar problems. Contributing to open-source projects is another excellent way to showcase your skills. It shows that you can work well with others understanding existing systems and contribute to the community. Plus, it's a great way to gain visibility and make connections with other engineers. We shall now speak about the résumés which work for you. Speaking about your resume, when it comes to landing a job as a machine learning engineer, your resume needs to be focused on real world projects and practical experience. So, be sure to highlight the projects that you've worked on, the tools you've used, and most importantly, the impact that your work has had. If you build a recommendation engine that boosted product sales by 20%, make sure that you include that. Employers want to see how your work contributes to solving real business problems. Don't forget to include links, Kaggle profile, or any other open source contributions that you have made. Employers love seeing code and showing them your projects is the best way to stand out. We shall now speak about what interviewers test in 2026. By now you must be wondering what do employers actually look for in an ML engineer. So in 2026 interviews are not just about technical knowledge. Employers just want to see how well you can communicate your thought process and real world problems. You'll likely face practical tests that challenge you to build a model or analyze data in real time. You'll also be tested on how well you explain your approach and justify the decisions that you made during the project. Prepare to discuss things like why you choose a specific algorithm for a problem or how you handle issues like data imbalance or overfitting along with what metrics you use to evaluate your model's performance. Being able to communicate clearly your process and how you arrived at your solution is a skill that will set you apart from other candidates. We shall now speak up about the ML trends that you will need to follow. Machine learning is evolving fast and staying up to date is key to remaining relevant in this field. So here are a few trends to keep an eye on in 2026. Generative models. These models can generate new data based on patterns they learn and they're used for things like text generation, image creation, and even music composition. AutoML automated machine learning tools are making it easier for non-experts to build machine learning models. As a result, more people will be able to contribute to this field without needing to become experts. Privacy first models. As privacy concerns grow, machine learning models are being designed to work securely and ethically and these don't compromise on user privacy. Staying on top of these trends will help you remain competitive and innovative in the ever evolving field of machine learning. So in conclusion, I will say consistency is key in machine learning. You don't need to know everything right away, but you must stay committed to learning and building. Start small, keep experimenting, and keep improving. The road to become a machine learning engineer may seem long, but with the right tools and mindset, you can get there. Thank you for watching and I'll see you in the next one. Thanks for watching the machine learning engineer road map for 2026. We hope that this video gave you a clear path to becoming a successful ML engineer. Ready to take the next step? Explore the Simply Learns professional certificate in AI and machine learning in partnership with Purdue University. Gain hands-on experience and industry recognized credentials to boost your career. >> Welcome to math refresher, probability and statistics. In this lesson, we are going to explain the concepts of statistics and probability. Describe conditional probability. Define the chain rule of probability. Discuss the measure of variance. Identify the types of gshian distribution. Basic of statistics and probability. Probability and statistics. Data science relies heavily on estimates and predictions. A significant portion of data science is made up of evaluations and forecast. Statistical methods are used to make estimates for further analysis. Probability theory is helpful for making predictions. Statistical methods are highly dependent on probability theory and all probability and statistics are dependent on data. Data is information acquired for reference or research via observations, facts, and measurements. Data is a set of facts structured in the form that computers can interpret such as numbers, words, estimations, and views. Importance of data. Data aids in seeing more about the information by identifying possible connections between two features. Data assists in the detection of distortion by uncovering hidden patterns based on prior information patterns. Data may be utilized to anticipate the future or predict the current state of affairs. Also, data aids in determining whether two pieces of information have any instance in common or not. Types of data. Data might be quantitative that is data that can be measured or counted in numbers or it may be qualitative which is data which is generally divided into groups or in simpler words which cannot be counted or measured in numbers. Let's consider an example. A customer information data of a bank may contain quantitative and qualitative data. Consider this snapshot where we have customer ID, surname, geography, gender, age, balance, has C or card is active member. Amongst these variables we can see surname is mostly qualitative as it cannot be counted and measured in numbers. Geography and gender are also qualitative as they cannot be counted in numbers and are mostly groups. has C or card that is has credit card and is active member although are containing numerical in form but these are categorical that means these have been divided into groups of one and zero that represent yes and no as an answer hence these two variables are also qualitative customer ID is again although a numerical data however the significance or intuition behind Customer ID is categorical. Hence, it may be kept in the qualitative data also. However, age and balance these are numerical information which have been measured or counted and numerical operations can be performed on them. Hence, these are under quantitative data categories. Introduction to descriptive statistics. Descriptive statistics. A descriptive measurement is summary measure that quantitatively portrays the most important features of a set of data allowing for a better comprehension of the information. Data can be measured as different levels. The levels of measurement describe the nature of information stored in the data assigned to the variables. Qualitative data can be measured as nominal or ordinal. Quantitative data can be measured in terms of interval and ratio type. Nominal data. The data is categorized using names, labels or qualities. For example, brand name, zip code, and gender. Ordinal data can be arranged in order or ranked and can be compared. Examples include grades, star reviews, position, and race, and date. Interval data is the data that is ordered and has meaningful differences between the data points. Example, temperature in Celsius and year of birth. Ratio data is similar to the interval level with the added property of inherent zero. Mathematical calculations can be performed on both interval as well as ratio data. For example, height, age, and weight. Population versus sample. Before analyzing the data, it's important to figure out if it's from a population or a sample. Population is a collection of all available items as well as each unit in our study. Sample is a subset of the population that contains only a few units of the population. Population data is used for study when the data pool is very small and can give all the required information. Samples are collected randomly and represent the entire population in the best possible way. Measures of central tendency. The central tendency is a single value that aids in the description of the data by determining its center position. Measures of central tendency are sometimes known as summary statistics or measures of central location. The most popular measurements of central tendency are mean, median, and mode. The normal distribution is a bell-shaped symmetrical distribution in which mean, median, and mode all are equal. The curve over here shows the bell-shaped curve or the normal distribution of variable X. The point over here that is X1 is the point which represents the mean, median and mode of this distribution. Mean mean is calculated by dividing these sum of all data values by the total number of data values. It gets affected when there are unusual or extreme values. It is sensitive to the outliers. Mean can be calculated as summation over all the values of X in a collection divided by the size of the collection. For example, we have a collection where we have values as 7 3 4 1 6 and 7. We find out the sum of these values which is 28 and there are total of six values. So 28 / 6 gives us a mean value of 4.66. Median, it is the middle value in the set of the data that has been sorted in ascending order. It is a better alternative to mean since it is less impacted by outliers and skewess. It is closer to the actual central value. Median is calculated differently for different sizes of data. Differentiated as if the total number of values is odd or if the total number of values is even. If the size of the data is odd. For example, in this case we have five elements. After sorting whatever middle value we get that means n + 1 by 2 term in this case 5 + 1 / 2 that is the third term which is four is the median value. In case when the total number of values is even like here there are six values. The average or the mean of the two central values is considered as the median. In this case the median is the mean of 6 and four which is five. Mode. Mode represents the most common value in the data set. It is not at all affected by extreme observations. It is the best measure of central tendency for highly skewed or non-normal distribution. Mode for categorical data is determined by estimating the frequencies for each categories and then the category with the highest frequency is considered to be mode. Like in this case seven has the highest frequency. Hence seven becomes the mode value. However, in case of continuous data or quantitative data, the calculation of mode is slightly different. The first step in calculation of mode is dividing the data into classes which are equal with then getting the frequency of data points lying in within that range of classes and finally selecting the class with the highest frequency. Using the range of that class and the frequencies, we can get the final mode value. Using the formula L plus FM minus F_sub_1 ultiplied to H / F minus F_sub_1 plus FM minus F_sub_2. Here L is the lower limit or the lower observation of the mode class. H is the size of the mode class. FM is the frequency of the mode class. F_sub_1 is the frequency of the class proceeding to mode and F_sub_2 is the frequency of the class succeeding to mode. This gives us the final mode value. Mean versus expectation. Now let's talk about mean versus expectation. So in general we use the expected value or expectation when we want to calculate the mean of a probability distribution that represents the average value we expect to occur before collecting any data. And mean on the other hand mean is basically used when we want to calculate the average value of a given sample. This represents the average value of raw data that we may have already collected. We can understand this by using a simple example. Now to calculate the expected value of this probability distribution, we can use a specific formula from the previous discussion. This is going to be the expected value where X is going to be the data value and this PX is the probability of value. For example, we could calculate the expected value for this probability distribution to be as shown. So here it will be 1.45 goals. So this represents the expected number of goals that the team will score in any given game. And then if you talk about calculating mean, so we typically calculate the mean after we have actually collected raw data. For example, suppose we record the number of goals that a soccer team will score in 15 different games. Now to calculate the mean number of goals scored per game, we can use the following formula where sum of x is basically the sum of all the goals divided by n and the number of records or we can say the sample size. It is as shown on the screen. So this represents the mean number of goals scored per game by the team. Measures of asymmetry. The difference between the three distinct curves can be studied in this image. The central curve is the normal or no skewess curve here. mean, median and mode all lie on the same point. This normal curve is symmetrical about its mean, median and mode. That means the left hand side of the curve is a mirror image of the right hand side of the curve. However, in case of negatively skewed data, the tail is elongated on the left hand side and the mean is smaller than the mode and the median values or is on the left hand side of the mode. Hence indicating that the outliers are in the negative direction. On the other hand, in case of positively skewed, the data is concentrated on the left hand side of the curve. While the tail is elongated or longer on the right hand side of the curve, the mean is greater than the mode and median or is on the right hand side of the mode and median indicating that the outliers are in the positive direction. Let's consider an example. The graph here shows the global income distribution for the year 2003 2013 and a projection for 2035. If we see the global income distribution statistics for 2003 it is highly right skewed. We can observe in the previous graph that in 2003 the mean of $3,451 was higher than the median of $1090. The global income is definitely not evenly distributed. The majority of people make less than $2,000 each year, while only a small percentage of the population earns more than $14,000. Measures of variability. Measures of variability. Dispersion. The measure of central tendencies provide a single value that addresses the full worth. However, the central tendency cannot depict the viewpoint entirely. The metric of dispersion helps us focus on the inconsistency in the data spread. Measures of dispersion describe the spread of the data. The range, intercortile range, standard deviation and variance are examples of dispersion measures. Range. The range of distribution is the difference between the largest and the smallest amount of data. The range, for example, does not include all of a series positive aspects. It concentrates on the most shocking aspects and ignores that aren't considered critical. For example, for a set 13, 33, 45, 67, 70, the range is 57. That is the maximum of this which is 70 minus the minimum over here which is 13. Variance. Variance is the average of all squared deviations. It is defined as the sum of squared distance between each point and the mean or the dispersion around the mean. The standard deviation is used as variance suffers from a unit difference. Variance can be computed as sigma square summation over x - mu^ 2 divided by n where mu is the mean of the data, x is the individual data point and n is the size of the data. This representation is for a population data. for a sample data variance can be computed as x minus xar whole square summation over it divided by n minus one. Here xbar is the mean of these sample data and n is the sample size. The units of values and variance are not equal. So another variability measure is used. Standard deviation. Standard deviation is a statistical term used to measure the amount of variability or dispersion around a mean. The standard deviation is calculated as the square root of variance. It depicts the concentration of the data around the mean of the data set. Standard deviation as indicated previously can be computed as square root of variance for a population data. Standard deviation sigma can be computed as square root of summation over x i minus mu^ square / n where mu is the mean of the data x i are the data points and n is the size. Let's consider an example. Let's find out the mean, variance, and standard deviation for this data. The data values are three, 5, 6, 9, and 10. To find out the mean, we first find the sum of all these data values that is 33 and divide it by the count, which is five. We get the mean of 6.6. To compute the variance, we start by computing the deviation. That is X minus the mean of X. Here 3 is one of the values of the data and 6.6 is the mean. So 3 - 6.6 squared and we do that to find out sum of all the deviations divided by the count which is five. We end up getting an overall variance of 6.64. Standard deviation as we know is measured at square root of variance that is square<unk> of 6.64 which amounts to 2.576. Measures of relationship. Measures of relationship coariance. Coariance is the measure of joint variability of two variables. It measures the direction of the relationship between the variables. It determines if one variable will cause the other to alter in the same way. Coariance between variable X and Y can be computed as summation over the product of X I - Xar and Y I - Y bar the whole divided by N minus one. Here Xar and Y bar are the mean of X and Y respectively. The value of covariance can range from minus infinity to a plus infinity. Correlation. Correlation is normalized coariance. It measures the strength of association between two variables. The most common measure for correlation is the Pearson correlation coefficient. Correlation between two variables X and Y can be measured with respect to coariance as coariance between X and Y divided by the standard deviation of X and standard deviation of Y. The value of correlation ranges from a negative 1 to positive 1. Types of correlation. Correlation can be either a positive correlation, zero correlation or a negative correlation. The first picture over here represents a perfect positive correlation wherein a straight line with a positive slope is representing the relationship between the two variables. Zero correlation means that the line representing the relationship between the two variables is horizontal to the xaxis. Perfect negative correlation can be represented by a straight line with a negative slope. Correlation equals to 1 implies a positive relationship. That is when one variable increases the other variable also increases. A correlation value of negative 1 implies a negative relationship. That is when one variable increases the other decreases. The correlation coefficient of zero shows that the variables are completely independent of each other. Let's consider an example. Here we have two variables height and weight. To compute the correlation between height and weight, we use the correlation formula as coariance of X and Y divided by standard deviation of X and standard deviation of Y. Here height is the X variable and weight is the Y variable. First to compute coariance we compute the x - xar and y - y bar values and then the product of them. We then compute x - xr² and y - y bar square values to compute the standard deviations of height and weight respectively. Correlation as we know has been defined as covariance of x and i and y divided by standard deviations of x and y. This can also be represented as summation over x - xr multiplied to y - y bar divided by square root of summation over sum of squared deviations that is x - xr square multiplied to square root of summation over y - y bar square that is sum of square deviations for y. Now let's find out values to put into this formula. First we find out the overall sum of height to get the mean of height which is 5.14. Similarly we get the sum of weight to get the mean of weight as 50. We now get the summation over x - xr multiplied to y - y bar to get the numerator for the formula. Then we compute x - xr square summation and y - y bar square that is sum of squared deviation of x and y respectively. Now we put in the values in this final correlation formula to get a correlation value of 0.889. This indicates that height and weight have a positive relationship. It is evident that as height grows, weight also increases. In this module, we will be talking about expectation and variance. So the expected value or we can say mean of a given variable that we can denote by X is a discrete random variable where it is a weighted average of the possible values that X can take and each value is going to be according to the probability of that specific event occurring. So usually the expected value of X is denoted by a simple formula where we can define the expectation based on the X parameter. which is going to be the sum of each possible outcome multiplied by the probability of the outcome occurring. So in more concrete terms, the expectation is what we would expect the outcome of an experiment to be on average. We can take an example for the coin. If a coin is being tossed 10 times, then one is most likely to get five heads and five tails. Same logic can be discussed if we talk about another example of rolling a die. So there are six possible outcomes when you roll a dieice 1 2 3 4 5 6 and each of these has a probability of 1 by 6 of occurring. So we can say that the expectation is going to be 1 multiplied by the probability of that happening which is going to be 1x 6 + 2x 6 + 3x 6 + 4x 6 + 5x 6 + 6x 6 and that is going to give us 3.5 as an output. The expected value is 3.5. So if you think about it, 3.5 is halfway between the possible values that I can take and this is what we should have expected. Next we talk about the concept of variance. So variance of a random variable allows us to know something about the spread of the possible values of the variable. So for a discrete random variable X, the variances of X is going to be denoted by using a simple formula that is going to be var equals E X - M the whole square where M is basically the expected value of the expectation of X. So this is more like a standard deviation of X which can also be represented by using this formula. So the variance does not behave in the same way as expectation when we multiply and add constants to random variables. So now there are two different type of variance that we can have a fair understanding on. First of all we have low variance and then we have high variance. So low variance simply means that there is a small variation in the production of the target function with changes in the trading data set and at the same time high variance as we can see here high variance shows a large variation in prediction of the target function with changes in the trading data set. So a model that shows high variance learns a lot and perform well with the training data set and it does not generalize well with the unseen data set and that's why as a result such a model gives good results with training data set but shows high error rates on the test data set and since the high variance a model learns too much from the data set it leads to an overfitting of the model. So model with high variance will be having couple of issues like it may lead to overfitting or it may also lead to increase in model complexities. Next we have skewess. So skewess in simple terms is basically a measure of asymmetry of a distribution. So distribution is asymmetrical when its left and right sides are not the mirror images. Right now this is a mirrored image and a distribution can have right positive or we can say negative or it can have zero skewess. So right skewed in this scenario is basically the distribution is longer on the right side of its peak and a left skew distribution is going to be we can say where it is longer on the left side. So we can see we have this one as a part of right side. It is more elongated towards the right side and this one is more elongated towards the left side. So we can think of skewess in terms of tails. A tail is long tampering and the end of a distribution. So it simply indicates that they are observations at one end of the distribution but that they are relatively infrequent. So a right skew distribution has a long tail on the right side as you can see here. So the number supports observed. Let's say we have a data on a per year basis. So again we can have a more skewess towards the right side where data is being dropping as we continue to increase the number of years. For example we may have a high sales towards the beginning of year suppose in 2022 but again as we proceed to 2023 second half we are seeing the dip in performance. So that is rightly skewed and same way let's suppose if we started with the sales figure it was really less in suppose 2002 but again as we proceeded to 2023 now our sales have been gradually increasing so it's more like skew towards the left section as a part of negative skew. Next we have curtosis. So curtosis is basically a measure of the tailness of a distribution. So tailness is how often the outliers occur and acts as courtesis. Is the tailness of the distribution related to a normal distribution. So a distribution with medium curtsis is called as meocurtic. A distribution with low curtosis like this one. This is called as the platicurtic and then distribution with high curtosis like this one. This is called as the leptocortic. So tails here they are tapering ends on either side of a distribution like this. So they represent the probability or the frequency of values that are extremely high or extremely low to the mean. In other words, tails here represents how often the outliers occur. So there are three type of curtosis. We have platocurtic which is negative, leptoccuric which is a positive towards the upper end and then we have messertic which is a normal distribution. So messertic is the medium tail. So normal distributions they have a curtosis of three. So any distribution with a curtsis of a prox value of three is going to be messertic. And curtosis is described in terms of excess curtises which is curtosis minus3. And since normal distribution they have a curtosis of three axis curtises makes comparing a distribution curtosis to a normal distribution even easier. Introduction to probability. Probability theory. Probability is a measure of the likelihood that an event will occur. Let's consider an example of coin toss where the chances of getting heads on a coin are 1 by two or 50%. The probability of each given event is between 0 and 1 both inclusive. Sum of an events cumulative probability cannot be greater than one. Hence the probability of an event x lies between zero and one. This means that the integral of probability of distribution over X equals to 1. Conditional probability. Conditional probability of any event A is defined as the probability of occurrence of A given that event B has previously occurred. Condition probability of event A given B can be estimated as probability of A intersection B that is probability of both A and B happening together divided by the probability of B. It is also written as that probability of A intersection B equals to probability of A given B multiplied to probability of B. Let's consider an example. In a coin, we are doing a two coin flip. Coin one gets heads, tails, heads, and tails in subsequent flips. while coin two gets tails, heads, heads, and tails in the subsequent flips. Now, the probability that coin one will get a head is 2 out of four. While the probability that coin two will get heads is again two out of four. The probability that both coin one and coin two will have a heads is just one out of the four flips. Hence the probability that coin one will get heads given that coin 2 is already heads can be computed as probability of coin one edge intersection coin 2 edge that is 1x4 divided by probability of coin 2 edge that's a given that is 2x 4 which is going to be 0.5 or 50% based base theorem Base theorem calculates the conditional probability of an event based on its prior probabilities. Basically base theorem incorporates the prior probability distribution to predict the posterior probabilities base theorem for conditional probability can be expressed as probability of A given B equals probability of B given A divided by probability of B multiplied to probability of A. Base theorem allows updating the probability values by using new information or evidence. Here probability of A is known as prior probability. That is the probability of event before any new data is collected. Probability of A given B is known as the posterior probability. It is the revised probability of an event occurring after taking into consideration the new information probability of B given A is known as the likelihood and probability of B is probability of observing an evidence B model. An example consider an example for calculating the likelihood of having diabetes based on frequency of fast food consumption. Here is the observed data. Let's say the fast food audience is 20%. Diabetes prevalence is 10% and 5% is fast food and diabetes. The chances of diabetes given fast food that is the conditional probability of D given B can be calculated as probability of diabetes and fast food together divided by probability of fast food. That means 5% divided by 20%. that equals 25%. Define an analysis can state eating fast food increases the chance of having diabetes by 25%. The multiplication rule of probability if events A and B are statistically independent and probability of A intersection B can be given as probability of A given B multiplied to probability of B. However, probability of A intersection B is also given as probability of A multiplied to probability of B. Here probability of A given B equals to probability of A when we assume that probability of B is non zero. Similarly, probability of B equals probability of B given A assuming probability of A is non zero. Chain rule of probability joint probability distributions over many random variables can be reduced into conditional distributions over a single variable. It can be expressed as probability of X1 X2 so on until Xn equals probability of X1 intersection probability of X I given probability of X1 till X I minus one. For example, the joint probability of A, B and C can be given as probability of A given B. C multiplied to probability of B given C multiply to probability of C. Logistic sigmoid. The logistics function is a type of sigmoid function that aims to predict the class to which a particular sample belongs. Its outcome is discrete binary value. a probability between zero and one. The logistic sigmoid is a useful function that follows the yes curve. It saturates when the input is very large or very small. Logistic sigmoid is expressed as sigma of x= 1 upon 1 + e to the power minus x. The logistic sigmoid can be expressed as sigmoid function of x is given as 1 upon 1 + e ^ min - x where e is the ooler's number. Gshian distribution. The gossian distribution is a type of distribution in which data tends to cluster around a central value with little or no bias to the left or right. It is often referred to as normal distribution. In absence of prior information, the normal distribution is frequently a fair assumption in machine learning equation. The formula for calculating Gaussian distribution is described as the normal distribution of X. That is the function of X given mean as mu and variance is sigma square can be calculated as 1 upon sigma square<unk> of 2<unk>i. E to the power minus/ X - mood divided by sigma square where mu is the mean or peak value which also is the expected value of X. Sigma is the standard deviation. Sigma square is the variance. A standard normal distribution has a mean of zero and a standard deviation of one. Gshian distribution can be univariate which describes the distribution of a single variable X. It can also be multivariate where it can just use to describe the distribution of several variables. It is represented in 3D of ND formats. Law of large numbers. Now let's talk about law of large numbers. The law of large numbers states that an observed sample average from a large sample will be close to the true population average and that it will get closer in the larger sample. So the law of large number does not guarantee that a given sample spatially a small sample will reflect the true population characteristics or that a sample does not reflect the true population will be balanced by a subsequent sample. This is for the law of large numbers to express the relationship between scale and growth rate. So there are multiple examples through which we can understand and it is widely used in statistical analysis in working with the central limit theorem in terms of the business growth. So there are multiple real time setup in which these are going to be used. So if you talk about tossing a coin, so tossing a coin in a number of times will give us two different type of outcomes. The result will spread evenly between head and tails and the expected average value is going to be half. That means 50 * tails and 30 * heads. But again, if you toss a coin 1,00 times, then the result can be in different manners because out of 1,000, let's say 850 times it has been head and only 150 times it has been tails and so on. So that's why the possibility of one event occurring is going to be changed in large sample sets as compared to a small sample sets as in let's say 10 times. So the number of heads and tails unbalanced for lower number of trials. So we can see it is unbalanced. But again as soon as we toss more number of coins more leans towards the balance value or we can see the observed averages. Next we have P value. So p value is basically a number calculated from the statistical test that describes how likely we are to have found a particular set of observations if the null hypothesis were true. So p values are used in hypothesis testing to help decide whether to reject the null hypothesis. And the smaller the p value, the more likely we are to reject the null hypothesis. So we have a term called as null hypothesis. So all statistical tests they have null hypothesis. So for most tests the null hypothesis is that there is no relationship between our variables of in first or that there is no difference among groups. For example in a two-tail t test the non-hypothesis is that the difference between two groups is going to be zero. So p value is going to tell us how likely it is that our data could have occurred under the null hypothesis. It is done by calculating the likelihood of a test statistic which is the number calculated by a statistical test using our data. So p value tell us how often we would expect to see a test statistic as extreme or more extreme than one calculated by a statistical test. if the null hypothesis of the test was true. So there are multiple limitations as well. So first one is the results can be significant but again they are they may not be practical as we have compared it can be based on multiple hypothesis for a game for the healthcare test. If the test is going to be positive or not it may show even values of the effect of a variable but not the magnitude in real life. What exactly is going to be the application of a drug test being failed in pharma company? Therefore, it is recommended to use confidence and levels in addition to the p values to quantify or we can say to give a solid figure to the reserve which we are going to get. The p values they are interpreted as supporting or we can say refuting the alternative hypothesis. So p value can only tell you whether or not the null hypothesis is supported. It cannot tell us whether our alternative hypothesis is true or why. So the risk of rejecting the null hypothesis is often higher than the p value. So especially when we are looking at a single study or when using small sample sizes. So this is because the smaller frame of reference, the greater are the chance that as we stumble across a statistically significant pattern completely by accident. Key takeaways. Key takeaways. Probability and statistics structure the premise of the data. The data helps in anticipating the future or gauging in view of the past patterns of information. The central tendency is a single value that helps to describe the data by identifying these central positions. The mean, median, and mode are the measures of central tendencies. The distribution where the data tends to be around a central value with a lack of bias or minimal bias towards the left or right is called as gshian distribution. >> My name is Richard Kersner with the simply learn team. That's get certified, get ahead. We're going to cover mathematics for machine learning. So today's agenda is going to cover data and its types. Then we're going to dive into linear algebra and its concepts, calculus, statistics for machine learning, probability for machine learning, hands-on demos, and of course throwing in there in the middle is going to be your matrixes and a few other things to go along with all this. Data and its types. Data denotes the individual pieces of factual information collected from various sources. It is stored, processed, and later used for analysis. And so we see here uh just a huge grouping of information, a lot of tech stuff, money, dollar signs, numbers uh and then you have your performing analytics to drive insights and hopefully you have a nice share your shareholders gathered at the meeting and you're able to explain it in something they can understand. So we talk about datas types of data we have in our types of data we have a qualitative categorical you think nominal or ordinal and then you have your quantitative or numerical which is discrete or continuous and let's look a little closer at those data type vocabulary always people's favorite is the vocabulary words okay not mine uh but let's dive into this what we mean by nominal nominal they are used to label various just uh label our variables without providing any measurable value. Uh country, gender, race, hair, color, etc. It's something that you either mark true or false. This is a label. It's on or off. Either they have a red hat on or they do not. Uh so a lot of times when you're thinking nominal data labels, uh think of it as a true false kind of setup. And we look at ordinal. This is categorical data with a set order or a scale to it. Uh and you can think of salary range is a great one. Uh movie ratings etc. You see here the salary range if you have 10,000 to 20,000 number of employees earning that rate is 150 20,000 to 30,000 100 and so forth. Some of the terms you'll hear is bucket. Uh this is where you have 10 different buckets and you want to separate it into something that makes sense into those 10 buckets. And so when we start talking about ordinal, a lot of times when you get down to the brass bones, again, we're talking true false. Uh so if you're a member of the 10 to 20k range, uh so forth, those would each be either part of that group or you're not. But now we're talking about buckets and we want to count how many people are in that bucket. Quantitative numerical data uh falls into two classes, discrete or continuous. And so data with a final set of values which can be categorized class strength questions answered correctly and runs hit in cricket. A lot of times when you see this you can think integer uh and a very restricted integer i.e. you can only have 100 questions um on a test. So you can it's very discreet. I only have a 100 different values that it can attain. So think usually you're talking about integers but within a very small range. They don't have an open end or anything like that. Uh so discrete is very solid, simple to count, set number. Continuous on the other hand uh continuous data can take any numerical value within a range. So water pressure, weight of a person etc. Usually we start thinking about float values where they can get phenomenally small in their in what they're worth. And there's a whole series of values that falls right between discrete and continuous. Um you can think of the stock market. You have dollar amounts. It's still discreet, but it starts to get complicated enough when you have like, you know, jump in the stock market from $525.33 to $580.67. There's a lot of point values in there. It'd still be called discreet, but you start looking at it as almost continuous because it does have such a variance in it. Now uh we talk about n we did we went over nominal and ordinal uh almost true false charts and we looked at quantitative and numerical data which we're starting to get into numbers. Discrete you can usually a lot of times discreet will be put into it could be put into true false but usually it's not. Uh so we want to address this stuff and the first thing you want to look at is the very basic which is your algebra. So we're going to take a look at linear algebra. You can remember back when your uklidian geometry. Uh we have a line. Well, let's go through this. We have linear algebra is the domain of mathematics concerning linear equations and their representations in vector spaces and through matrices. I told you we're going to talk about matrices. Uh so a linear equation is simply um uh 2x + 4 y - 3 z = 10. Very linear. 10 x + 12.4 4 y = z. And now you can actually solve these two equations by combining them. Uh, and that's we're talking about a linear equation. And the vectors we have a + b equals c. Now we're starting to look at a direction. And these values usually think of an xyz plot. Um, so each one is a direction. And the actual distance of like a triangle A is C. And then your matrix can describe all kinds of things. Um, I find matrixes uh confuse a lot of people, not because they're particularly difficult, but because of the magnitude and the different things are used for. And a matrix is a chart or a um, you know, think of a spreadsheet, but you have your rows and your columns. And you'll see here we have a * b= c. Very important to know your counts. Uh, so depending on how the math is being done, what you're using it for, making sure you have the same rows and number of columns or a single number, there's all kinds of things that play in that that can make matrixes confusing. Uh, but really has a lot more to do with what domain you're working in. Uh, are you adding in multiple polomials where you have like uh uh ax^2 plus b y plus, you know, you start to see that can be very confusing versus a very straightforward matrix. And let's just go a little deeper into these because these are such primary this is what we're here to talk about is these different math uh mathematical computations that come up. So we're looking at linear equations. Let's dig deeper into that one. An equation having a maximum order of one is called a linear equation. Uh so it's linear because when you look at this we have uh ax plus b= c which is a one variable. We have uh two variable ax plus b y= c ax plus b y plus z cz= d and so forth. But all of these are to the power of one. You don't see x squ. You don't see x cubed. So we're talking about linear equations. That's what we're talking about. And they're addition. If you have already dived into say neural networks, you should recognize this ax plus b y plus cz um setup plus the intercept. uh which is basically your your neural network each node adding up all the different inputs and we can drill down into that most common formula is your y = mx plus c. So you have your uh y equals the m which is your slope, your x value plus c which is your um y intercept. They kind of labeled it wrong here threw me for a loop but the the c would be your y intercept. So when you set x equal to zero, y equals c. And that's that's your y intercept right there. Uh and that's they they just had reversed value of y. When x equals 0 equals the y intercept, which is c and your slope gradient line, which is your m. So you get your y = 2x + 3. And there's lots of easy ways to compute this. This why this is why we always start with the most basic one when we're solving one of these problems. And then of course the um one of the most important takeaways is the slope gradient of the line. Uh so the slope is very important that m value. Uh in this case we went ahead and solved this. If you have y = 2x + 3 you can see how it has a nice line graph here on the right. So matrixes a matrix refers to a rectangular representation of an array of numbers arranged in columns and rows. So we're talking m rows by n columns here. A11 is denotes the element of the first row in the first column. Similarly a12 and it's really pronounced a11 in this particular setup. So it's row one column one. A12 is a of row one column 2 uh first row and second column and so on. And there's a lot of ways to denote this. I've seen these as like a capital letter A, smaller case A for the top row or I mean you can see where they can go all kinds of different directions as far as the value. You just take a moment to realize there needs to be some designation as far as what row it's in and what column it's in. And we have our uh basic operations. We have addition. So when you think about addition, you have uh two matrices of 2x two and you just add each individual number in that matrix and then when you get to the bottom you have uh in this case the solution is 12. 10 + 2 is 12 5 + 3 is 8 and so on. And the same thing with subtraction. Now again you're counting matrices. You want to check your um dimensions of the matrix. The shape you'll see shape come up a lot in programming. And we're talking about dimensions. We're talking about the shape. If the two shapes are equal, this is what happens when you add them together or subtract them. And we have multiplication. When you look at the multiplication, you end up with a very uh slightly different setup going. Now, if we look at our last one, we're um uh we're like, why this always gets to me when we get to matrices. They don't really say why you multiply matrices. Um, you know, my first thought is 1 * 2, 4 * 3. But if you look at this, we get 1 * 2 + 4 * 3, 1 * 3 + 4 * 5, uh, 6 * 2 + 3 * 3, 6 * 3 + 3 * 5. If you're looking at these matrices, uh, think of this more as an equation. And so we have, uh, if you remember when we back up here for our multiple line equations, let's just go back up a couple slides where we were looking at a two variable. So this is a two variable equation. ax plus b y= c. Um and this is a way to make it very quick to solve these variables. And that's why you have the matrix and that's why you do the multiplication the way they do. And this is the dotproduct of uh 1 * 2 + 4 * 3 1 * 3 + 4 * 5 uh 6 * 2 + 3 * 3 6 * 3 + 3 * 5. Then he gives us a nice little 14 23 21 and 33 over here which then can be used and reduced down to a simple um formula as far as solving the variables as you have enough inputs. Uh and then in matrix operations when you're dealing with a lot of matrices uh now keep in mind multiplying matrices is different than finding the product of two matrices. Okay? So we're talking about multiplication we're talking about solving uh for equations. When you're finding the product, you are just finding one times two. Keep that in mind because that does come up. I've had that come up a number of times where I am altering data and I get confused as to what I'm doing with it. Uh transpose flipping the matrix over it's diagonal. Comes up all the time where you have you still have 12, but instead of it being uh 128, it's now 1214 821. You're just flipping the columns and the rows. Uh and then of course you can do an inverse um changing the signs of the values across this main diagonal. And you can see here we have the inverse a to the minus1 and ends up with uh instead of 12 8 14 12 it's now -22 -2 vectors uh vector just means we have a value and a direction and we have down four numbers here on our vector. Uh in mathematics a one-dimensional matrix is called a vector. Uh so if you have your xplot and you have a single value that values along the x- axis and it's a single dimension. If you have two dimensions you can think about putting them on a graph. You might have x and you might have y and each value denotes a direction. And then of course the actual distance is going to be the hypothesis of that triangle. Uh and you can do that with three dimensionals x y and z. uh and you can do it all the way to nth dimensions. So when they talk about the k means uh for categorizing and how close data is together, they will compute that based on the Pythagorean theorem. So you would take uh the square of each value, add them all together and find the square root and that gives you a distance as far as where that point is, where that vector exists or the actual point value. And then you can compare that point value to another one and it makes a very easy comparison versus comparing uh 50 or 60 different numbers. And that brings us up to gene vectors and I gene values. Uh I gene vectors the vectors that don't change their span while transformation and I gene values the scalar values that are associated to the vectors. Conceptually you can think of the vector as your picture. you have a picture. It's um uh two dimensions x and y. And so when you do those two dimensions and those two values or whatever that value is um that is that point but the values change when you skew it and so if we take and we have a vector a and that's a set value uh b is um your is your you have a and b which is your hygiene vector. Two is the hyene value. So, we're altering all the values by two. That means we're u maybe we're stretching it out one direction, making it tall if you're doing picture editing. Um that that's one of the places this comes in. But you can see when you're transforming uh your different information, how you transform it is then your hygiene value. And you can see here uh vector after line transition uh we have 3 a is the gene vector. Three is the gene value. So A doesn't change. That's whatever we started with. That's your original picture. And three uh is skewing it one direction and maybe uh B is being skewed another direction. And so you have a nice tilted picture because you've altered it by those by the values. So let's go ahead and pull up a demo on linear algebra. And to do this, I'm going to go through my trusted Anaconda into my Jupiter notebook. and we'll create a new uh notebook called linear algebra. Since we are working in Python, uh we're going to use our numpy. I always import that as np or numpy array. Probably the most popular um module for doing matrixes and things in given that this is part of a series. I'm not going to go too much into numpy. Uh we are going to go ahead and create two different variables. A for a numpy array 10 15 and b 29. We'll go ahead and run this. And you can see there's our two arrays 10 15 29. And I went ahead and added a space there in between so it's easier to read. And since it's the last line, we don't have to put the print statement on it unless you want. We can simp but we can simply do a + b. So when I run this, uh, we have 105 29 and we get 30 24, which is what you expect. 10 + 20 15 + 9. You could almost look at this addition as being um just adding up the columns on here coming down. And if we wanted to do it a different way, we could also do a t plus b. T. Remember that t flips them. And so if we do that, we now get them uh we now have 30 24 going the other way. We could also do something kind of fun. There's a lot of different ways to do this. uh as far as a plus b I can also do a plus b t and you're going to see that that will come out the same the 30 24 whether I transpose a and b or transpose them both at the end and likewise we can very easily subtract two vectors I can go a minus b and we run that and we get - 106 now remember this is the last line in this particular section that's why I don't have to put the print around it Um, and just like we did before, we can transpose either the individual or we can transpose the main setup and then we get a - 106 going the other way. Now, we didn't mention this in our notes, but you can also do a scalar multiplication. Let me just put down the scaler so you can remember that. Uh what we're talking about here is I have uh this array here u and if I go a time u uh we'll take the value two we'll multiply it by every value in here. So 2 * 30 is 60 2 * 15 and just like we did before um this happens a lot because when you're doing matrices you do need to flip them you get 6030 coming this way. So in numpy uh we have what they call dotproduct and uh what this this in a twodimensional vectors it is the equivalent of two matrix multiplication and remember we were talking about matrix multiplication uh where it is the well let's walk through it we'll go ahead and start by defining two um numpy arrays we'll have uh 10 20 256 or our u and our E uh and then we're going to go ahead and do if we take the values uh and if you remember correctly an array like this would be 10 * 25 + 20 * 6. We'll go ahead and uh print that. There we go. And then we'll go ahead and do the uh np dot of u comma v. And we'll find when we do this, we go and run this uh we're going to get uh 370 370. So this is a strain multiplication where they use it to solve uh linear algebra uh when you have multiple numbers going across. And so this could be very complicated. We could have a whole string of different variables going in here. But for this we get a nice uh value for our dot multiplication. And we did um addition earlier which was just your basic addition. Uh and of course a matrix you can get very complicated on these or in this case we'll go ahead and do um let's create two complex matrixes. This one is a matrix of um you know 1210 46 431. We'll just print out A so you can see what that looks like. Here's print A. We print A out. You can see that we have a um 2x3 layer matrix for A. And we can also put together always kind of fun when you're playing with print values. Uh we could do something like this. We could go in here. There we go. Uh, we could print a. We have it end with uh equals a run. And this kind of gives it a nice look. Uh, here's your matrix. That's all this is. Comma n means it just tags it on the end. That's all all that is doing on there. And then we can simply add in what is a plus b. And you should already guess because this is the same as what we did before. There's no difference. Uh, we do a simple vector addition. We have 12 + 2 is 14. 10 + 8 is 18. And so on. And just like we did the uh matrix addition, we can also do a minus b and do our matrix subtraction. And we look at this uh we have what? 12 - 2 is 10. 10 - 8 um where are we? Oh, there we go. 8 min confusing what I'm looking at. I should have reprinted out the original numbers. Uh but we can see here 12 - 2 is of course 10. 10 - 8 is 2. Uh 4 - 46 is - 42 and so forth. So same as a subtraction as before, we just call it matrix subtraction. It's identical. Now if you remember up here, we had uh scalar addition where we're adding just one number to a matrix. You can also do scalar multiplication. Uh and so simply if you have a single value A and you have B which is your array, we can also do A * B. When we run that, uh, you can see here we have 2 * 4 is 8. Uh, 5 * 4 is 20 and so forth. You're just multiplying the four across each one of these values. And this is an interesting one that comes up. A little bit of a brain teaser is matrix and vector multiplication. And so when we're looking at this, uh, we are just do a regular arrays. It doesn't necessarily have to be a numpy array. We have A which has our um array of arrays and B which is a single array. And so we can from here do the dot AB and this is going to return two values. And the first value is that it's you could say it's like uh um we're doing the this array B array first with A and then with a second one. And so it splits it up. So you have a matrix of vector multiplication and you can mix and match. When you get into really complicated uh backend stuff, this becomes more common because you're now you got layers upon layers of data and so you you'll end up with a matrix and a set of uh vector matrices. Do you want to multiply? Now keep in mind that if you're doing data science a lot of times you're not looking at this. This is what's going on behind the scenes. So if you're in um the scikit looking at sklearn where you're doing linear regression models, this is some of the math that's hidden behind the scenes that's going on. Other times you might find yourself having to do part of this and manipulate the data around so it fits right and then you go back in and you run it through the scikit. And if we can do um up here where we did a uh matrix and vector multiplication, we can also do matrix to matrix multiplication. And if we run this where we have the two matrices, uh you can see we have a very complicated array that of course comes out on there for our dot. And just to reiterate it, we have our transpose a matrix which is your T. And so if we create a matrix A and we do transpose it, you can see how it flips it from 5 10 15 20 25 30 to 5 15 25 10 2030 uh rows and columns. And certainly with the math, uh, this comes up a lot. Um, and it also comes up a lot with XY plotting. When you put it into piplot, you have one format where they're looking at pairs of numbers and then they want all of X's and all Y's. So, you know, the transpose is an important tool both for your math and for plotting and all kinds of things. Another tool that we didn't discuss uh is your identity matrix. Uh and this one is more definition. Uh the identity matrix um we have here one where we just did uh two. So it comes down as one 0 0 1 uh 1 0 0 1 0. It creates a diagonal of one. And what that is is when you're doing your identities, you could be comparing all your different features to the different features and how they correlate. And of course when you have uh feature one compared to feature one to itself it is always one uh where usually it's between zero one depending on how well correlates. So when we're talking about identity matrix that's what we're talking about right here is that you create this preset matrix and then you might adjust these numbers depending on what you're working with and what the domain is. And then another thing we can do uh kind of wrap this up. we'll hit you with the most complicated uh um piece of this puzzle here is an inverse um a matrix. And let's just go ahead and put the um it's a lengthy description. Let's go and put the description. This is straight out of the uh the website for um Numpy. Uh so given a square matrix A, here's our square matrix A, which is 2 1 0 0 1 0 1 2 1. Keep in mind 3x3 it's square. It's got to be equal. It's going to return the matrix A inverse satisfying dot A um A inverse. So here's our matrix multiplication. Um and then of course it equals the dot uh yeah a inverse of a um with an identity shape of u a dotshaped zero. This is just reshaping the identity. That's a little complicated there. Uh so we go and have our here's our array. Uh we'll go ahead and run this. And you can see what we end up with is we end up with uh an array 0.5 minus.5 and so forth with our 211 going down 2100 0 1 0 1 2 1. Uh getting into a little deep on the math. Understanding when you need this is probably really is is what's really important when you're doing data science versus uh handwriting this out and looking up the math and handwriting all the pieces out. You do need to know about the linear algorithm inverse of a. Uh so if it comes up you can easily pull it up or at least remember where to look it up. You took a look at the algebra side of it. Let's go ahead and take a look at the calculus side of uh what's going on here with the machine learning. So calculus, oh my goodness. and differential equations. You got to throw that in there because that's all part of the bag of tricks especially when you're doing large neural networks but also comes up in many other areas. The good news is most of it's already done for you in the back end. Uh so when it comes up you really do need to understand from the data science not data analytics. Data analytics means you're digging deep into actually solving these math equations. U and a neural network is just a giant differential equation. Uh so when we talk about calculus uh we're going to go ahead how and understand it by talking about cars versus time and speed. Uh so helps to calculate the spontaneous rate of change. Uh so suppose we plot a graph of the speed of a car with respect to time. So as you can see here going down the highway probably merged into the highway from an on-ramp. So I had to accelerate so my speed went way up. uh stuck in traffic, merged into the traffic. Traffic opens up and I accelerate again up to the speed limit and uh maybe it peters off up there. So you can look at this as as um the speed versus time. I'm getting faster and faster because I'm continually accelerating and if I hit the brakes it go the other way. So the rate of change of speed with respect to time is nothing but acceleration. How fast are we accelerating? The acceleration is the area between the start point of x and the end point of delta x. Uh so we can calculate a simple if you had x and delta x, we could put a line there and that slope of the line is our acceleration. Now that's pretty easy when you're doing linear algebra, but I don't want to know it just for that line and those two points. I want to know it across the whole of what I'm working with. That's where we get into calculus. So when we talk about the distance between x and delta x it has to be the smallest possible near to zero in order to approximate the acceleration. Uh so the idea is that instead of I mean if you ever did took a basic calculus class they would draw bars down here and you would divide this area up um let's go back up a screen. you divide this area of this time period up into maybe 10 sections and you use that and you could calculate the acceleration between each one of those 10 sections kind of thing. Uh and then we just keep making that space smaller and smaller until delta x is almost uh infantismally small. And so we get a function of a uh equals a limit as h goes to zero of a function of a plus h minus a function of a over h. And that is you're computing the slope of the line. We're just computing that slope under smaller and smaller and smaller samples. Uh and that's what calculus is. Calculus is the integral. You can see down here we have our nice uh integral sign. Looks like a giant s. And that's what that means is that we've taken this down to as small as we can for that sampling. Uh so we're talking about calculus. for finding the area under the slope is the main process in the integration. Similar small intervals are made of the smallest possible length of x plus delta x where delta x approaches almost an infantismly small space. And then it helps to find the overall acceleration by summing up all the lengths together. Uh so we're summing up all the accelerations from the beginning to the end. And so here's our integral. we sum of a of x * d ofx= a + c. Uh that is our basic calculus here. So when we talk about multivvariant calculus, uh multivariate calculus deals with functions that have multiple variables. And you can see here we start getting into some very complicated equations. Um uh change in w over change of time equals change of w over change of z. the differential of z to dx, differential of x to dt, it gets pretty complicated. Uh, and it really translates into the multivariate integration using double integrals. And so you have the the sum of the sum of f ofx of y of d of a equals the sum from c to d and a to b of f ofxy dx dy equals uh the sum of a to b sum of c to d of fxy dy dx. understanding the very specifics of everything going on in here and actually doing the math is usually calculus one, calculus 2, and differential equations. Uh so you're talking about three fulllength courses to dig into and solve these math equations. What we want to take from here is we're talking about calculus. Uh we're talking about summing of all these different slopes. And so we're still solving a linear uh expression. We're still solving y = mx + b, but we're doing this for infantismally small x's. And then we want to sum them up. That's what this integral sign means. The the sum of a of x d of x= a plus c. And when you see these very complicated uh multivariate differentiation using the chain rule uh when we come in here and we have the change of w to the change of t equals the change of w dz uh and so forth. That's what's going on here. That's what these means. We're basically looking for the area under the curve which really comes to how is the change changing. Speed's going up. How is that changing? And then you end up with a multiple layer. So if I have three layers of neural networks, how is the third layer changing based on the second layer changing which is based on the first layer changing? And you get the picture here that now we have a very complicated uh multivariate integration um with integrals. The good news is we can solve this uh mathematically and that's what we do when you do neural networks and reverse propagation. Uh so the nice thing is that you don't have to solve this on paper unless you're a data analysis and you're working on the back end of integrating these formulas and building the script to actually build them. So we talk about applications of calculus. Uh it provides us the tools to build an accurate predictive model. Um so it's really behind the scenes we want to guess at what the change of the change of the change is. That's a little goofy. I I know I just threw that out there. It's kind of a meta term. But if you can guess how things are going to change, then you can guess what the new numbers are. Multivariat calculus explains the change in our target variable in relation to the rate of change in the input variables. So there's our multiple variables going in there. If uh one variable is changing, how does it affect the other variable? And then in gradient descent, calculus is used to find the local and global maxima. And this is really big. Uh we're going actually going to have a whole section here on gradient descent because it is really I mean I talked about neural networks and how you can see how the different layers go in there but gradient descent is one of the most key things for trying to guess the best answer to something. So let's take a look at the code behind gradient descent. And uh before we open up the code, let's just do real quick uh gradient descent. Let's say we have a curve like this. And most common is that this is going to represent your error. Oops. Error. There we go. Error. Uh hard to read there. And I want to make the error as low as possible. And so what I'm looking at it is I want to find this line here which is the minimum value. So we're looking for the minimum and it does that by uh sampling there and then it based on this it guesses it might be someplace here and it goes hey this is still going down. It goes here and then goes back over here and then goes a little bit closer and it's just playing a high low until it gets to that spot, that bottom spot. And so we want to minimize the error. And uh on the flip note, you could also want to be maximizing something. You want to get the best output of it. Uh that's simply minus the value. Uh so if you're looking for where the peak is, this is the same as a negative for where the valley is and looking for that valley. Uh that's all that is. And this is a way of finding it. So the cool thing is um all the heavy lifting is done. Um I actually ended up putting together one of these a while back as uh when I didn't know about sidekick and I was just starting. Boy, it's a long while back. And uh is playing high low. How do you play high low? not get stuck in the valleys, uh, figure out these curves and things like that. Well, you do that and the back end is all the calculus and differential equations to calculate this out. The good news is you don't have to do those. Uh, so instead, we're going to put together the code and let's go ahead and see what we can do with that. So, uh, guys in the back put together a nice little piece of code here, which is kind of fun. uh some things we're going to note and this is this is really important stuff because when you start doing your data science and digging into your machine learning models uh you're going to find these things are stumbling blocks. Uh the first one is current x where do we start at? Uh keep in mind your model that you're working with is very generic. So whatever you use to minimize it the first question is where do we start? Um, and we started at this cuz the algorithm starts at x= 3. So, we arbitrarily picked five. Learning rate is uh how many bars to skip going one way or the other. Uh, in fact, I'm going to separate that a little bit because these two are really important. Um, if we're dealing with something like this where we're talking about um uh well, here's our here's the function. We're going to use our um gradient of our function um 2 * x + 5. Keep it simple. So that's a function we're going to work with. So if I'm dealing with increments of a thousand, 0.1 is going to be a very long time. And if I'm dealing with increments of 0.001, uh 0.1 is going to skip over my answer. So I won't get a very good answer. Um and then we look at precision. This tells us when to stop the algorithm. So again, very specific to what you're working on. uh if you're working with money and you don't convert it into a float value, uh you might be dealing with 0.01, which is a penny. That might be your precision you're working with. Um and then of course the previous step size, max iterations, uh we want something to cut out at a certain point. Usually that's built into a lot of minimization functions. And then here's our actual uh formula we're going to be working with. And then we come in, we go while previous step size is greater than precision and its is less than max itters. Say that 10 times fast. Um we're just saying if it's uh if we're if we're still greater than our precision level, we still got to keep digging deeper. Um and then we also don't want to go past a thou or whatever this is, a million or 10,000 uh running. That's actually pretty high. Um, we almost never do max iterations more than like a 100 or 200. Rare occasions you might go up to four or 500 if it's depending on the problem you're working with. Uh, so we have our previous equals our current. That way we can track timewise. Uh, the current now equals the current minus the rate times the formula of our previous x. So now we've generated our new version. Uh, previous step size equals the absolute current previous. Uh, so we're looking for the change in x errors equals iterations plus one. That's how we know to stop if we get too far. And then we're just going to print the local minimum occurs at x on here. And if we go ahead and run this, uh, you can see right here, it gets down to this point and it says, hey, um, local minimum is minus 3.3222 for this particular series we created. Uh, and this is created off of our formula here. Lambda x2 * x + 5. Now, when I'm running this stuff, uh, you'll see this come up a lot and uh, with the sklearn kit and and one of the nice reasons of breaking this down the way we did is I could go over those top pieces. Uh, those top pieces are everything when you start looking at these minimization toolkits in built-in code. And so from um we'll just do it's actually docs.ip.org and we're looking at the scikit. There we go. Um optimize minimize. You can only minimize one value. You have the function that's going in. This function can be very complicated. Uh so we used a very simple function up here. It could be there's all kinds of things that could be on there. And there's a number of methods to solve this as far as how they shrink down. Uh and your x knot. There's your there's your start value. So your function, your start value. Um there's all kinds of things that come in here that we can look at which we're not going to. Um optimization automatically creates constraints bounds. Some of this it does automatically, but you really the big thing I want to point out here is you need to have a starting point. You want to start with something that you already know is mostly the answer. Uh if you don't, then it's going to have a heck of a time trying to calculate it out. Or you can write your own little script that does this and and does a high low guessing and tries to find the max value. That brings us to statistics. What this is kind of all about is figuring things out. Lot of vocabulary and statistics. Uh so statistics, well, I guess it's all relative. It's definitely not an ed class. Uh so a bunch of stuff going on. Statistics. Statistics concerns with the collection, organization, analysis, interpretation, and presentation of data. That is a mouthful. Um, so we have from end to end we're come from, is it valid? What does it mean? How do we organize it? Um, how do we analyze it? Then you got to take those analysis and interpret it into something that uh people can use, kind of reduce it to understandable. Um, and nowadays you have to be able to present it. If you can't present it, then no one else is going to understand what the heck you did. So when we look at the terminologies, uh, there is a lot of terminologies depending on what domain you're working in. So clearly if you're working in um, a domain that deals with viruses and tea cells and and how does you know where does that come from? And you're studying the different people. then you're going to have a population if you are working with um mechanical gear um you know a little bit different if you're looking for the wobbling statistics uh to know when to replace a rotor on a machine or something like that uh that can be a big deal. You know we have these huge fans that turn in our sewage processing systems and so those fans they start to wobble and hum and do different things that the sensors pick up at one point. Do you replace them instead of waiting for it to break, in which case it costs a lot of money. Instead of replacing a bushing, you're replacing the whole fan unit. Uh an interesting project that came up for our city a while back. Uh so population, all objects are measurements whose properties are being observed. Uh so that's your population, all the objects. It's easy to see it with people because we have our population and large. Um but in the case of the sewer fans, we're talking about how many the fan units. That's the population of fans that we're working with. You have a parameter, a matrix uh that is used to represent a population or characteristic. You have your sample, a subset of the population studied. You don't want to do them all because then you don't have a if you come up with a conclusion for everyone, you don't have a way of testing it. So you take a sample. Uh sometimes you don't have a choice. You can only take a sample of what's going on. you can't um study the whole population and a variable a metric of interest for each person or object in a population. Types of sampling we have probabilistic approach uh selecting samples from a larger population using a method based on the theory of probability and we'll go into a little bit more deeper on these. We have random, systematic, stratified and then you have nonprobabilistic approach selecting samples based on the subjective judgment of the researcher rather than random selection. Uh it has to do with convenience trying to reach a quota um or snowball. Uh and they're very biased. That's one of the reasons you'll see this big stamp on it says biased. Uh so you got to be very careful on that. So probabilistic sampling. Uh when we talk about a random sampling, we select randomsiz samples from each group or category. So we it's as random as you can get. Uh when we talk about systematic sampling, we're selecting randomsiz samples from each group or category with a fixed periodic interval. Uh so we kind of split it up. This would be like a time setup or different categories. And you might ask your question, what is a category or a group? Uh if you look at I'm going to go back a window. Let's say we're studying um economics of different of an area. Um we know pretty much that based on their culture where they came from, they might need to be separated. And so uh and when I say separated, I don't mean separated from their their uh place where they live. I mean as far as the analysis, we want to look at the different groups and make sure they're all represented. So if we had like an 80% uh of a group that is uh say Hispanic and or Indian and also in that same area we have 20% 20% who are let's call our expatriots that left America and they're nice and uh your Caucasian group. We might want to sample a group that is representative of both. Uh so we're talking about stratified sampling and we're talking about groups. Those are the groups we're talking about. And that brings us to stratified sampling. Selecting approximately equal-sized samples from each group or category. Uh this way we can actually separate the categories and give us an insight into the different cultures and how that might affect them in that area. Uh so you can see these are very very different kind of depends on what you're working with um as far as your data and what you're studying. And so we can see here just to go a little bit more, we'd have selecting 25 employees from a company of 250 employees randomly. Don't care anything about them, what groups are in, which office are in, nothing. Um, and we might be selecting one employee from every 50 unique employees in a company of 250 employees. And then we have selecting one employee from every branch in the company office. So we have all the different branches. There's our group or our categories by the branch. And the category could depend on what you're studying. So it has a lot of variation on there. You see this kind of grouping and categorizing is also used to generate a lot of misinformation. Uh so if you only study one group and you say this is what it is then everybody assumes that's what it is for everybody. And so you got to be very careful of that and it's very unethical thing to kind of do. So types of statistics uh we talk about statistics we're going to talk about descriptive and inferential statistics. There are so many different terms in statistics to break it up. Uh so we so we're talking about a particular setup. So we're talking about descriptive and inferential uh statistics. You the base of the word describe is pretty solid. You're describing the data. What does it look like? With inferial statistics, we're going to take that from the small population to a large population. So if you're working with a drug company, uh you might look at the data and say these people were helped by this drug. They did uh 80% better as far as their health or 80% better survival rate than the people um who did not have the drug. So we can infer that that drug will work in the greater populace and will help people. So that's where you get your inferential. Uh so we are predicting how it's going to affect the greater population. So descriptive statistics it is used to describe the basic features of data and form the basis of quantitative analysis of data. So we have a measure of central tendencies. We have your mean, median and mode. And then we have a measure of spread like your range, your interquartile range, your variance and your standard deviation. And we're going to look at all these a little deeper here in a second. Uh but one of them you can think of is um how the data difference differences you know what's the max min range all that stuff is your spread and anything that's just a single number is usually your central uh tendencies measure of central tendencies. So we talk about the mean. It is the average of the set of values considered. What is the average outcome of whatever is going on? And then your median separates the higher half and the lower half of data. Uh so where's the center point of all your different data points? So your mean might have some a couple really big numbers that skew it uh so that the average is much higher than if you took those outliers out where the median would by separating the high from the low might give you a much lower number. You might look at and say, "Oh, that's that's odd. Why is the average so much higher than the median?" Well, it's because you have some outliers or why is it so much lower? And then the mode is the most frequent appearing value. Uh this is really interesting. If you're studying economics and how people are doing, you might find that the most common um income like in the US was at one point 24,000 a year where the average was closer to 80,000 and it's like wow, what a difference. Well, there's some people have a lot of money and so that skews that way up. So the average person is not making that kind of money. And then you look at the median income and you're like well the median income is a little bit closer to the average. Uh so it does create a very interesting way of looking at the data. Again these are all uh central tendencies single numbers you can look at for the whole spread of the data. And we look at the measure of central tendencies. The mean is the average marks of a students in a classroom. So here we have the mean sum of the marks of the students total number of students. And as we talked about the median, uh if we have zero through 10 and we take half the numbers and put them on one side of the line, half the numbers on the other side of the line, uh we end up with five in the middle. And then the mode, what mark was scored by most of the students in a test. In a simple case where most people scored like an 82% and got certain problems wrong, easy to figure out. uh not so easy when you have different areas where like you have like the um oh let's go back to economy a little bit more difficult to calculate if you have a large group that scores that makes 30,000 and a slightly bigger group that makes 26,000. So what do you put down for the mode? Uh certainly there's a number of ways to calculate that and there's actually a different variations depending on what you're doing. So now we're looking at a measure of spread uh range. What's the difference between the highest and the lowest value? First thing you want to look at, you know, it's we had everybody in the test scored between 60 and 100%. Somebody got 100% or maybe 60 to 90%. It was so hard that a lot of people could not get 100%. Um, and you have your interquartile range. Quartortiles divide a rankordered data set into four equal parts. Very common thing to do as part of all the basic packages. Whether you're working in uh data frames with pandas, whether you're working in scala, whether you're working in R, um you'll see this come up where they have range, your min, your max, and then it'll have your interquartile range. How does it look like in each quarter of data? Variance measures how far each number in the set is from the mean and therefore from every other number in the set. uh so you have like a how much turbulence is going on in this data and then the standard deviation it is the measure of the variance or the dispersion of a set of values from the mean and you'll usually see uh if I'm doing a graph I might have the value graphed um and then based on the the error I might graph graph the standard deviation and the error on the graph as a background so you can see how far off it is uh so standard deviation is used a lot so measurement a spread uh marks of a student out of a 100. Uh we have here from 50 to 63 or 50 to 90. Uh so the range maximum marks minimum marks we have 90 to 45 and the spread of that is 45 90 - 45 and then we have the interquartile range using the same marks over there. You can see here where the median is and then there's the first quarter, the second quarter and the third quarter based on splitting it apart by those values. And to understand the variance and standard deviation, we first need to find out the mean. Uh so here's our our you know calculating the average there. We end up at approximately 66 for the average. And then we look at that and the variance once we know the means we can do equals the marks minus the mean squared. Why is this squared? Uh because one, you want to make sure it's you don't have like if you if you're putting all this stuff together, you end up with an error as far as one's negative, one's positive, one's a little higher, one's a little lower. Uh so you always see the squared value and over the total observations. And so the standard deviation equals the square root of the variance, which is approximately 16. And if you were looking at um a predictable model, you would be looking at the deviation based on the error. How much error does it have? Uh that's again really important to know if you're if your prediction is predicting something, what's a chance of it being way off or just a little bit off. Now that we've looked at the um tools as far as some of the basics for doing your statistics and what we're talking about, let's go ahead and pull up a little demo and show you what that looks like in Python code. Uh so you can get some little hands-on here. For that, let's go back into our Jupyter notebook in Python. Now, almost all of this you can do in Numpy. Last time we worked um in Numpy. This time we're going to go ahead and use pandas. And if you remember from pandas on here, uh, this is basically a data frame, rows, columns. Let's just go ahead and do a print df.head and run that. And you can see we have uh the name Jane, Michael, William, Rosie, Hannah, and their salaries on here. And of course instead of having to do all those hand calculations and add everything together and divide by the total we can do something very simple on this uh like use the command mean in pandas. And so if I go ahead and do this print df pick our column salary because we want to find the means of that colery. We want to find the means of that column. Uh and we go and print this out and you can see that the uh average income on here is 71,000. Uh, and let's just go ahead and do this. We'll go ahead and put in uh means. And if we're going to do that, we also might want to find the median. And the median is uh very similar except it actually is just median. Uh we're used to means and average. It's kind of interesting that those are they use the two different words. Uh there can be in some computations slight differences but for the most part the means is the average. Uh and then the median oops let's put a median here. DF salary that way it displays a little better. We can see the median is 54 um000. So the halfway mark is significantly below the average. Why? Because we have somebody in here who makes 189,000. Darn you Rosie for throwing off our numbers. Uh but that's something you'd want to notice. This is this is the difference between these is huge and so is what is the meaning behind that when you're studying a populace and looking at uh the different data coming in. And of course we also want to find out hey what's the most uh common income that people make in this little tiny sample. And so we'll go ahead and do the mode. And you can see here with the mode uh it's at 50,000. So this is this is very telling that most people are making 50,000. The middle point is at 54,000. So half the people are making more than that. What that tells me is that if the most common income is way is below the median, then there's a few there's a you know there's a a lot of high salaries going up, but there's some really low salaries in there. And so this trend which is very common in statistic you when you're in analyzing the economy and different people's income is pretty common and the bigger difference between these is also very important when we're studying statistics. Uh and when you hear someone just say hey the average income was you might start asking questions at that point. Why aren't you talking about the median income? Why aren't you talking about the mode the most common income? What are you hiding? Uh and if you're doing these analysis, you should be looking at these saying, "Hey, why why are this discrepancies? Why are these so different?" And of course, with any uh analysis, it's important to find out the minimum and the maximum. So, we'll go ahead. It's just simply uh um min'll pull up your minimum and then do max pulls up the maximum. pretty straightforward on as far as um translating it and knowing what your you know what the your lowest value and what your highest value is here. Um which you'll use to generate like a spread later on. And real quick on no mode uh note that it puts mode zero. Like I said, there's a couple different ways you can compute the mode. Um although you know the standard one's pretty good. We can of course do the range which is your max minus your min. So now we have a range of 149,000 between the upper end and the lower end. And you might want to be looking up the individual values on all of these. But it turns out there is a describe feature in pandas. And so in pandas we can actually do df salary describe. And if we do this you can see we have that there's seven uh setups. Here's our mean. Um, our standard deviation, which we didn't compute yet, which would just be a std. And you got to be a little careful because when it computes it, it looks for axes and things like that. Uh, we have our minimum value, and here's our cortiles, uh, our maximum value, and then of course the name salary. Uh, so these are the these are the basic statistics. You can pull them up and like just describe. This is a dictionary. So I could actually do something like um in here I could actually go uh count and run. And now it just prints the count. Uh so because this is a dictionary, you can pull any one of these values out of here. It's kind of a quick and dirty way to pull all the different information and then split it up depending on what you need. Now if I just walked in and gave you this information um in a meeting, at some point you would just kind of fall asleep. That's what I would do anyway. Um, so we want to go ahead and and see about graphing it here. And we'll go ahead and put it into a histogram and plot that graph on it of the salaries. And let's just go ahead and put that in here. So we do our mattplot inline. Remember that's a Jupiter's notebook thing. Uh, a lot of the new version of the mattplot library does it automatically, but just in case I always put it in there. Uh, import mattplot library piplot as plt. That's my plotting. And then we have our data frame. Uh I don't I guess I really don't need to respell the data frame. Maybe we could just remind oursel what's in it. So we'll go ahead and just uh print DF. That way we still have it. And then we have our salary. DF salary salary.plot history title salary distribution color gray. Uh plot AXV line salary the mean value. So, we're going to take the mean value um color violet line style dash. This is just all making it pretty. Uh what color dash line width of two that kind of thing and the median. And let's go ahead and run this just so you can see what we're talking about. And so up here we are taking on our plot. Um so here's the data. Here's our our data frame printed out so you can see it with the salaries. We're looking at the salary distribution and just look at this the way they're the salary is distributed. Um you have our in this case we did let's see we had red for the median. We have violet for our average or mean and you can just see how it really I mean here's our outlier. Here's our person who makes a lot of money. Here's the um average and here's the median. Um and so as you look at this you can say wow um based on the average it really doesn't tell you much about what people are really taking home. All it does is tell you how much money is in this you know what the average salary is. So some of the things you want to take away in addition to this is that it's very easy to plot um an AXV line. These are these up and down lines for your markers. Um, and as you display the data, I mean, you can add all kinds of things to this and get really complicated. Keeping it simple is pretty straightforward. I look at this and I can see we have a major outlier out here. We can definitely do a histogram and stuff like that. Um, but you know, picture's worth a thousand words. What you really want to make sure you take away is that we can do a basic describe which pulls all this information out and we can print any of the individual information from the describe uh because this is a dictionary. And so if we want to go ahead and look up um the mean value, we can also do describe mean. So if you're doing a lot of statistics, uh being able to doesn't have the print on there, so it's only going to print um the last one, which happens to be the mean. Uh you can very easily reference any one of these. And then you can also, if you're doing something a little bit more complicated and you don't need just the basics, you can come through and pull any one of the individual um references from the from the pandas on here. So now we've had a chance to describe our data. Uh let's get into inferential statistics. Inferential statistics allows you to make predictions or inferences from data. And you can see here we have a nice little picture movie ratings and um if we took this group of people and said hey how many people like the movie dislike it can't say and then you ask just a random person who comes out of the movie who hasn't been in this study uh you can infer that 55% chance of saying liked 35% chance of saying disliked or a 10 or 11% chance of can't say. So that that's real basics of what we're talking about is you're going to infer that the next person is going to follow these statistics. Uh so let's look at point estimation. Uh it is a process of finding an approximate value for a population's parameter like mean or average from random samples of the population. Let's take an example of testing vaccines for COVID 19. Uh vaccines and flu bugs, all that. It's a pretty big thing of how do you test these out and make sure they're going to work on the populace. A group of people are chosen from the population. Medical trials are performed. Results are generalized for the whole population. So here's a protected here's our small group up here where we've selected them. We run medical trials on them and then the results work for the population. You nice diagram with the arrows going back and forth and the very scary co virus in the middle of one. And let's take a look at the applications of inferial statistics. Very central is what they call hypothesis testing uh and the confidence interval which go with that. And then as we get into probability, we get into our binomial theorem, our normal distribution and central limit theorem. Hypothesis testing. Hypothesis testing is used to measure the plausibility of a hypothesis assumption by using sample data. Now when we talk about theorems, theory, hypothesis, uh keep in mind that if you are in a philosophy class, theory is the same as hypothesis where theorem is a scientific uh statement that is something that has been proven although it is always up for debate because in science we always want to make sure things are up to debate. So a hypothesis is the same as a phil philosophical class calling a theory where theory in science is not the same. Theory in science says this has been well proven. Gravity is a theory. Uh so if you want to debate the theory of gravity try jumping up and down. If you want to have a theory about why the economy is collap collapsing in your area that is a philosophical debate. Very important. I've heard people mix those up and it is a pet peeve of mine. When we talk about hypothesis testing, the steps involved in hypothesis testing is first we formulate a hypothesis. We figure out the right test to test our hypothesis. We execute the test and we make a decision. And so when you're talking about hypothesis, you're usually trying to disprove it. If you can't disprove it and it works for all the facts, then you might call that a theorem at some point. So in a use case, uh let's consider an example. We have four students. were given a task to clean a room every day. Sounds like working with my kids. They decided to distribute the job of cleaning the room among themselves. They did so by making four chits which has their names on it and the name that gets picked up has to do the cleaning for that day. Rob took the opportunity to make chits and wrote everyone's name on it. So here's our four people, Nick, Rob, Emlia, Imlia, and Summer. Now Rick, I imlia and summer are asking us to decide whether Rob has done some mischief in preparing the chits i.e whether Rob has written his name on one of the chit for that we will find out the probability of Rob getting the cleaning job on first day second day third day and so on till 12 days the probability of Rob getting the job decreases every day i.e his turn never comes up then definitely he has done some mischief while making the chits. So the probability of Rob not doing work on day one is uh three out of four there's a 75 chance that he didn't do work. Uh 2 days 34 * 34 equals.56. 3 days you have 34 34 34s which equals42. Uh when you get to day 12 it's 0032 which is less than 0.05. Remember this 0.05 uh that comes up a lot when we're talking about um certain values when we're looking at statistics. Rob is cheating as he wasn't chosen for 12 consecutive days. That's a very high probability when on day 12 he still hasn't gotten the job cleaning the room. So we come up to our important important terminologies. We have null hypothesis. A general statement that states that there is no relationship between two measured phenomenon or no assoc association among the groups. Alternative hypothesis contrary to the null hypothesis it states whenever something is happening a new theory is preferred instead of an old one. And so the two hypothesis go hand in hand. Uh so your null this is always interesting in in when talking about data science and the math behind it. It's about proving that the things have no correlation. Null hypothesis says these two have zero relation to each other. Where the alternative hypothesis says hey we found a relation. This is what it is. We have p value. The p value is a probability of finding the observed or more extreme results when the null hypothesis of a study question is true. And the t value it is simply the calculated difference represented in units of standard error. The greater the magnitude of t, the greater the evidence against the null hypothesis. And you can look at the t value as being specific to the test you're doing where the p value is derived from your t value and you're looking for what they call the 5% or the 0.05 showing that it has a high correlation. So digging in deeper, let's assume that a new drug is developed with the goal of lowering the blood pressure more than the existing drug. And this is a good one because uh the null value here isn't that you don't have any drug. The null value here is that it's better than the existing drug. The new drug doesn't lower the blood pressure more than the existing drug. Now if we get that uh that says our null hypothesis is correct. There is no correlation and the new drug is not doing its job. The alternative hypothesis the new drug does significantly lower the blood pressure more than the existing drug. Uh, yay, we got a new drug out there. And that's our alternative hypothesis or the H1 or HA. And we look at the p value results from the evidence like medical trials showing positive results which will reject the null hypothesis. And again, they're looking for um a 0.05 or 5%. And the t value comparing all the positive test results and finding means of different samples in order to test hypothesis. So this is specific to the test. how uh what percentage of increase did they have and this leads us to the confidence intervals. Uh a confidence interval is a range of values we are sure our true values of observations lie in. Let's say you asked a dog owner around you and asked them how many cans of food do you buy for your uh per year for your dog? Through calculations you got to know that the on average around 95% of the people bought around 200 to 300 cans of food. Hence we can say that we have a confidence interval of 230 where 95% of our values lie in that spread data spread. Uh and this the graph really helps a lot. So you can start seeing what you're looking at here where you have the 95% you have your peak. In this case it's a normal distribution. So you have the nice bell curve equal on both sides. It's not asymmetrical. And 95% of all the values lie within a very small range. And then you have your outliers the 2.5% going each way. So we touched upon hypothesis. Uh we're going to move into probability. Uh so you have your hypothesis. Once you've generated your hypothesis, we want to know the probability of something occurring. Probability is a measure of the likelihood of an event to occur. Any event can be predicted with total certainty and can only be predicted as a likelihood of its occurrence. So any event cannot be predicted with total certainty. It can only be predicted as a likelihood of its occurrence. Uh score prediction. how good you're going to do in whatever sport you're in, weather prediction, stock prediction, if you've studied physics and chaos theory, even the location of the chair you're sitting on has a probability that it might move 3 ft over. Granted, that probability is one in like uh I think we calculated as under one in trillions upon trillions. So, it's the better the probability, the more likely it's going to happen. There are some things that have such a low probability that we don't see them. So we talk about a random variable. Uh random variable is a variable whose possible values are numerical outcomes of a random phenomena. So uh we have the coin toss. How many heads will occur in the series of 20 coin flips? Probably you know the on average there are 10, but you really can't know because it's very random. How many times a red ball is picked from a bag of balls if there's equal number of of red balls and blue balls and green balls in there. How many times the sum of digits on two dice uh result or five each? Um so you know there's how often you going to roll two fives on your pair of dice. So in a use case uh let's consider the example of rolling two dice. We have a random variable outcome equals y. You can take values 2 3 4 5 6 7 8 9 10 11 12. So we have a random variable and a combination of dice and instead of looking at how many times um both dice were roll five let's go ahead and look at a total sum of five and you have in as far as your random variables you can have a one four equals 5 4 1 2 3 32 so four of those roles can be four if you look at all the different options you have four of those random rolls can be a five and if we look at the total number which happens to be 36 different options. Uh you can see that we have four out of 36 chance every time you roll the dice that you're going to roll a total of five. You're going to have an outcome of five. And uh we'll look a little deeper as to what that means. Uh but you could think of that at what point if someone never rolls a five or they always roll a five, can you say, "Hey, that person's probably cheating." uh we'll look a little closer at the math behind that but let's just consider this as one of the cases is rolling two dice and gambling. There's also a binomial distribution. It is the probability of getting success or failure as an outcome in an experiment or trial that is repeated multiple times and the key is is by meaning two binomial. Uh so passing or failing an exam, winning or losing a game and getting either head or tails. So if you ever see binomial distribution, it's based on a um true false kind of setup. You win or lose. Let's consider a use case and let's consider the game of football between two clubs, Barcelona and Dortmund. The teams will have to play a total of four matches and we have to find out the chances of Barcelona winning the series. So we look at the total games and we're looking at five different games or matches. Let's say that the winning chance for Barcelona is 75% or 75. That means at each game they have a 75% chance that they're going to win that game and losing chances are 25% or 0.25. Clearly 75 plus.25 equals 1. So that accounts for 100% of the game. Probability for getting K wins in n matches is calculated. And we we're talking like so if you have five games uh and you want to know if I play um how many wins in those five games should I get? What's a percentage on those? And the probability for getting k wins and n matches is calculated by px= k= n c p the k q to the n minus k. Here p is the probability of success and q is the probability of failure. And so we can do total games of n= 5 where k equals 012 345 p which is the chance of winning is 75 q the chance of losing equals 1 - p which equals 1 - 0075 which equals.25 the probability that Barcelona will lose all of the matches can then just plug in the numbers and we end up with a009765625. So very small chance they're going to lose all their matches. And we can plug in uh the value for two matches. Probability that Barcelona will win at least two matches is 00878. And of course we can go on to probability that Barcelona will win three matches the 26 and of course four matches and so on. And it's always nice to take this information um and let let's find the cumulative discrete probabilities for each of the outcomes where Barcelona has won three or more matches x= 3 x= 4 x= 5 and we end up with the p =264 plus 395 + 237 which equals 89. In reality the probability of Barcelona winning the series is much higher than 75. And it's always nice to uh put out a nice graph so you can actually see the number of wins to the probability and how that pans out with our binomial case. Continuing in our important terminology, location, the location of the center of the graph depends on the mean value. And uh this is some very important things. So much of the data we look at and when you start looking at probabilities almost always has a normalized look like the graph in the middle. uh but you do have left skewed where the data is skewed off to the left and you have more stuff happening off to the left and you have right skewed data and so when this comes up and these probabilities come up where they're skewed it's really important to take a closer look at that uh mostly you end up with a normalized set of data but you got to also be aware that sometimes it's a skewed data and then the height height of the slope inversely depends upon the standard deviation so you can see down here the standard deviation is really large it kind of squished es it out. And if the standard deviation is small, then most of your data is going to hit right there in the middle. You're going to have a nice peak. Um, and so being aware of this that you might have a probability that fits certain data, but it has a lot of outliers. So you're if you have a really high standard deviation, um, if you're doing stock market analysis, this means your predictions are probably not going to make you much money. uh where if you have a very small deviation, you might be right on target and set to become a millionaire. Which leads us to the zcore. Zcore tells you how far from the mean a data point is. It is measured in terms of standard deviations from the mean. Around 68% of the results are found between one standard deviation. Around 95% of the results are found between two standard deviations. And you read the symbols. Of course, they love to throw some Greek letters in there. we have mu minus 2 sigma. Mu is just a quick way. It's that kind of funky u. It just means the mean. Uh and then the sigma is the standard deviation. And that's the o with a little arrow off to the right or the little waggly tail going up. The o with a with a line on it. Uh so mu minus 2 sigma is your uh 95% of the results are found between two standard deviations. Central limit theorem. This goes back to the skew. If you remember, we were looking at the skew values on this previous slide. Have left skewed, normalized, and right skewed. When we're talking about it being skewed or not skewed, the distribution of the sample means will be approximately normally distributed, evenly distributed, not skewed. If you take large random samples from the population with the mean mu and the standard deviation sigma with replacement and you can see here um uh of course we have our uh mu minus 2 sigma and the spread down here the mean the median and the mode and so when you're talking about very large populations these numbers should come together and you shouldn't have a skewed value. If you do that's a flag that something's wrong. That's why this is so important to be aware of what's going on with your data, where your samples are coming from, and the math behind it. And if you're going to do all this, we got to jump into conditional probability. The conditional probability of an event A is the probability that the event will occur given the knowledge that an event B has already occurred. And you'll see this as B theorem. B- A S B. Uh, and this is read. I mean, you have these funky looking little P brackets. A B. This is the probability of A being true while B is already true. And you have the probability of B being true when A is already true. So, P B of A probability of A being true divided by the probability of B being true. And we talk about BA's theorem which occurred back in the 1800s when he discovered this. This is such an important formula and it's really it's not if you actually do the math you could just kind of do um um XY equals J K and then you divide them out and you're going to see the same math but it works with probabilities which makes it really nice. And so if you have a you might have uh eight or nine different studies going on in different areas different people have done the studies they brought them together. Um if we look at today's co virus the virus spread uh certainly the studies done in China versus the studies the way they're done in the US that data is different in each of those studies but if you can find a place where it overlaps where they're studying the same thing together you can then compute the changes that you need to make in one study to make them equal and this is also true if you have a study of uh um one group and you want to find out more about it. So this formula is very powerful. Uh it really has to do with the data collection part of the math and data science and understanding where your data is coming from and how you're going to combine different studies in different groups. And we'll go ahead and go into a use case. Uh let's find out the chance of a person getting lung disease due to smoking. Uh and this is kind of interesting the way they word this. Um let's say that according to medical report provided by the hospital states that around 10% of all patients they treated suffered lung lung disease. Uh so we have kind of a generic medical report. They further found out uh by a survey that 15% of the patients that visit them smoke. So we have 10% that are lung disease and um 15% of the patients smoke. And finally, 5% of the people continued smoke even when they had lung disease. Uh not the brightest choice um but you know it is an addiction so it can be really difficult to kick. And so we can look at the probability of a uh prior probability of 10% people having lung disease. And then probability b probability that a patient smokes is 15%. uh and the probability of B um if B then A the probability of a patient smokes even though they have lung disease is 5%. And probability of A is B probability that the patient will have lung disease if they smoke. And then when you put the formulas together uh you get a nice solution here. You get the probability of A of B probability that the patient will have lung disease if they smoke. And you can just plug the numbers right in and we get a 3.33% chance. Hence, there is a 3.33% chance that a person who smokes will get a lung disease. So, we're going to pull up a little Python code. Always my favorite. Roll up the sleeves. Keep in mind we're going to be doing this um kind of like the backend way so that you can see what's going on. And then later on we're going to create um we'll get into another demo which shows you some of the tools that are already pre-built for this. Let's start by creating a set. So we're going to create a set with curly braces. This means that our set has um only unique values. So you have a list uh you have your tupils which can never change and then you have um in this case the the set. So 47, you can't create a 47, 4. It'll delete the four out. So it's only unique values. And if you use dictionaries, quick reminder, this should look familiar because it is a dictionary uh where you have a value and that value is assigned to or that key is assigned to a value. Uh so you could have a key value set up as a dictionary. So it's like a dictionary without the value. It's just the keys and they all have to be unique. And if we run this, we have a set of 47. We can also take a list, a regular um setup. And I'm going to go ahead and just throw in another number in here, four, and run it. Uh, and you can see here if I take my list 1 2 3 4, and I convert it to a set, and here it is. My set from list equals set my list. The result is 1 2 3 4. So, it just deletes that last four right out of there. And with the sets, you can also go in there and um print here is my set. My set uh three is in the set. And then if you do three in my set, that's going to be a logic function. Uh and one in my set, six is not in the set, and so forth. And if we run this, we get uh three is in the set true one is in the set false because 357 is another one. Six is in the set uh six is not in the set. So not in my set. You can also use this with a list. We could have just used 357 and it would have um the same response on there is three in. Usually you do if three is in but three in my set is still works on just a regular list. We'll go ahead and do a little iteration. We're going to do kind of the dice one. Remember, um, uh, 1 2 3 4 5 6. And so, we're going to bring in an iteration tool and import product as product. And, uh, I'll show you what that means in just a second. So, we have our two dice. We have dice A and it's going to be a set of values. Um, they can only have one value for each one. That's why they put it in a set. And if you remember from range, it is up to seven. So this is going to be 1 2 3 4 5 six. It will not include the seven. And the same thing for our dice B. And then we're going to do is we're going to create a list which is the product of A and B. So what's um A plus B? And if we go ahead and run this uh it'll print that out. And you'll see um in this case when they say product cuz it's an iteration tool, we're talking about creating a tupole of the two. So we've now created a tupole of all possible outcomes of the dice where dice A is 1 2 3 1 six and dice B is 1 to six. And you can see 1 one two one to three and so forth. You remember we had a slide on this earlier where we talked about um the different all the different outcomes of a dice. We can play around with this a little bit. Uh we can do in dice equals 2 divi dice faces 1 2 3 4 5 6. Uh another way of doing what we did before and then we can create an event space where we have a set which is the product of the dice faces repeat equals end dice. And we'll go ahead and just run this. And you can see here it just again puts it through all the different possible variables we can have. And then if we wanted to take the same uh set on here and print them all out like we had before uh we can just go through for outcome and event space. Outcome end equals. So the event space is creating a sequence. And as you can see here when we print it out it stacks them versus going through and putting them in a nice line. And we'll go ahead and do something. Um, let's go print. Since we have the end printing with a comma, that just means it's just going to it's not going to hit the return going down to the next line. Uh, and we'll go ahead and do the length of our event space. Uh, that'll be an important variable we're going to want to know in a minute. And of course, if I get carried away with my typing of length, uh, we'll print it twice and it'll give me an error. Uh so we have 36 different possible variations here and we might want to calculate something like um what about the multiple of three? What if we want to have uh the probability of the multiple of three in our setup? And so uh we can put together the code for the outcome in event space of xy equals outcome if x + y remainder 3. So, we're going to divide by three and look at the remainder and it equals zero. Then it's a favorable outcome and we're going to pop that outcome on the end there. And we'll turn it into a set. So, the favor outcome equals a set. Not necessary uh because we know it's not going to be repeating itself, but just in case, we'll go ahead and do that. And if we want to print out the outcome, we can go ahead and see what that looks like. And you can see here these are all u multiples of three. Uh 1 plus 2 is 3, 5 + 4 is 9, which divided by 3 is 3, and so forth. And just like we looked up the length uh of the one before, let's go ahead and print the length of our f outcome so we can see what that looks like. There we go. And of course, I did forget to add the print in the middle because we're looping through and putting an end on the on the setup on there. So, we're going to put the print in there. And if I run this, you can see uh um we end up with 12. So, we have 36 total options. Uh we have 12 that are multiple that um add up to a multiple of three. And we can easily conver compute the probability of this uh by simply taking the length of our favorable outcome over the length of the event space. And if we print it out, let me put that in there. Probability last line. So we just type it in. We end up with a 3333 chance. And that's roughly a third. And we might want to make this look nice. So let's go ahead and put in another line there. The probability of getting the sum which is a multiple of three is 3333. We can compute the same thing for five dice. And if we do this for five dice and go ahead and run it. Um you can see we just have a huge amount of choices. So it just goes on and on down here. And we can look at the uh length of the event space. And we have over 7,776 choices. That's a lot of choices. And if we want to ask the question like we did above, uh what is the sum where the sum is a multiple of five but not a multiple of three, we can go through all of these different options. And then uh you can see here uh d1 d2 d3 d4 d5 equals the outcome. And if uh you add these all together and the division by five does not have a remainder of zero but the remainder is also of a division by three is not equal to zero. So the multiple of five is equal to zero but the multiple of three is not. We can just appin that on here and then we can look at that uh favorable outcome. We'll go ahead and set that and we'll just take a look at this. What's our length of our favorable outcome? It's always good to see what we're working with. And so we have 94 out of 776. And then of course we can just do a simple division to get the probability on here. What's the probability that we're going to roll a multiple of five when you add them together, but not a multiple of three? And so, we're just going to divide those two numbers. And you can see here we get.116255 or 11.62%. And so, you can really have a nice visual that this is not really complicated math right here on probabilities. uh it's just how many options do you have and how many of those are you possibly going to be able to um come up with with the solution you're looking for. And this leads us to a confusion matrix. A confusion matrix is a table which is used to describe the performance of a classification model on a set of test data for which the true values are known. And so you'll see on the left we have the predicted and the actual and we have a negative uh false negative positive true positive um and then we have false positive and true negative. And you can think of this as your predicted model. What does that mean? That means if you divided your data and you use twothird of it to create the model, you might then test it against an actual case for the last third to see how well it comes out. How many times was it uh true positive versus uh false positive? It gave a false positive response. And you can imagine in medical uh situations, this is a pretty big deal. You don't want to give a false positive. So you might adjust your model accordingly so you don't have a false positive. Say with a co virus test, it'd be better to have a false negative and they go back and get retested than to have 30% false positives where then the test is pretty much invalid. So in a use case uh like cancer prediction, let's consider an example where a cancer prediction model is put to the test for its accuracy and precision. Actual result of a person's medical report is compared with the prediction made by the machine learning model. And so you can see here here's our actual predicted uh whether they have cancer or not. You know cancer a big one. You don't want to have a u false positive. I mean a false negative. In other words, you don't want to have it tell you that you don't have cancer when you do. So that would be something you'd really be looking for in this particular domain. You don't want a false negative. Uh and this is again, you know, you've created a model, you have hundreds of people or thousands of pieces of data that come in. There's a real famous case study where they have the imagery and all the measurements they take and there's about 36 different measurements they take. And then if you run the a basic model, you want to know just how accurate it is. How many um negative results do you have that are either telling people they have cancer that don't or telling people that don't have cancer that they do? And then we can take these numbers and we can feed them into our accuracy, our precision and our recall. Uh so accuracy, precision and recall, accuracy metric to measure how accurately the results are predicted. And this is your um total um true where you got the right results. you add them together, the true positive, the true negative over all the results. So what percentage of them were accurate versus what were wrong? We talk about precision is a metric to measure how many of the correctly predicted cases are actually turned out to be positive. Uh so we have a precision on true positive. Again, if you're talking about like uh COVID testing with the viruses, uh you really want this to be a a high number. you want this true um that to be the center point where you might have the opposite if you're dealing with cancer where you want no false negatives. Uh so this is your metric on here. Precision is your test positive uh true positive plus uh false positive. And then your recall how many of the actual positive cases we were able to predict quickly with our model. Uh so test positive is the test positive plus the false negative on there. and we'll want to go ahead and do a demo on the naive bay classifier. Before I get too far into uh naive bay classifier because we're going to pull it from the sklearn or the uh scikit, um let's go ahead kind of an interesting page here for classifiers. When you go into the sklearn kit, there's a lot of ways to do classification. And I'll just zoom up in here so you can see some of the titles. Uh there's everything from the nearest neighbor linear uh but we're going to be focusing on the naive bays over here. And this is just um a sample data set that they put together. And you can see how some of these have a very different output. The naive bay remember is set up as probably the most simplified uh calculator or um set of predictions out there. And so what we've been talking about with the true false and stuff like that where there's a uh an belief that there is a independent assumption between the features where the features are very assumed to have some kind of connection uh then we can go ahead and use that for the prediction. And so that's what we're using as a naive bay classifier versus many of the other classifiers that are out there. For this we're going to use uh the social network ads. It's a little data set on here and let me go and just open that up the file. Uh here we go. It has user ID, gender, age, estimated salary, uh purchased. And so we have you can see the user ID male 19 uh estimated salary 19,000 and purchased zero. Uh so it's either going to make a purchase or not. So look at that last one 01. We should be thinking of binomials. we should be thinking of a simple naive base classifier kind of setup. So if we close this out, we're going to go ahead and import our numpy as np. We're nice to have a a good visual of our data. So we'll put in our mattplot library. Here's our pandas, our data frame. Uh and then we're going to go ahead and import the data set. And the data set's going to be we're going to read it from the social network ads.csv. Then we're going to print the head just so you can see it again uh even though I showed you it in the file. And X equals the data set I location uh two three values and Y is going to be the four uh column 4. Let me just run this so it's a little easier to go over that. Um you can see right here we're going to be looking at uh 012 is age and estimated salary. So 2 three and that's what I location just means um that we're looking at the number versus a regular location. Uh regular location you'd actually say age and estimated salary. And then column four is did they make a purchase? They purchased something. Uh so those are the three columns we're going to be looking at when we do this. And we've gone ahead and imported these and imported the data. So now our data set is all set with this information in it. And we'll need to go ahead and split the data up. Uh so we need our from the sklearn model selection we can import train test split. Uh this does a nice job. We can set the random state so it randomly picks the data. And we're just going to take uh 25% of it is going to go into the test our x test and our y test and the 75% will go to x train and y train. And that way once we create our model, we can then have data to see just how accurate or how well it has performed with our um prediction. The next step in pre-processing our data is to go ahead and do feature scaling. Now, a lot of this is start to look familiar. If you've done a number of the other modules and setup, you should start noticing that we bring in our data. We take a look at what we're working with. uh we go ahead and split it up into training and testing. Uh in this case, we're going to go ahead and scale it. Scale it means we're putting it between a value of minus one and one uh or someplace in that middle ground there. This way, if you have any huge set, you don't have this huge um setup. If we go back up to here where salary salary is 20,000 versus age 35, well, there's a good chance with a lot of the backend math that 20,000 will skew the results and the estimated salary will have a higher impact than the age instead of balancing them out and letting the calculations weigh them properly. And finally, we get to actually create our naive bay model. Um, and then we're going to go ahead and import the Gazian naive bay. And the Gazian is is uh the most basic one. That's what we're looking at now. It turns out though, if you go to the SK um learn kit, uh they have a number of different ones you can pull in there. There's a um Bernoli. I'm not I've never used that one. Categorical um compliment, and here's our Gazian. Uh so there's a number of different options you can look at. Gazian when you come to the naive bay is the most commonly used. Uh so we're talking about the naive bays that's usually what people are talking about when they when they're pulling this in. And one of the nice things about the gauian if you go to their website um to sklearn the naive bay gauian there's a lot of cool features. One of them is you can do partial fit on here. Um that means if you have a huge amount of data, you don't have to process it all at on you once you can batch it into the Gausian uh NB model. And there's many other different things you can do with it as far as fitting the data and how you um manipulate it. We're just doing the basics. So we're going to go ahead and create our classifier. We're going to equal the Gazian NB and then we're going to do a fit. We're going to fit our training data and our training solution. So, X-Rain, Y train, and we'll go ahead and run this. Uh, it's going to tell us that it it ran the code right there. And now we have our trained classifier model. So, the next step is we need to go ahead and run a prediction. We're going to do our y predict equals the classifier.predict x test. So, here we fit the data and now we're going to go ahead and predict. And now we get to our confusion matrix. Uh so from the sklearn matrix metrics you can import your confusion matrix just as saves you from doing all the simple math. It does it all for you. And then we'll go ahead and create our confusion metrics with the y test and the y predict. So we have our actual and we have our predicted value. And you can see from here this is the chart we looked at. Here's predicted. So, true positive, false positive, false negative, true negative. And if we go ahead and run this, there we have it. 653725. And in this particular uh prediction, we had 65 uh or predicted the truth as far as a a purchase. They're going to make a purchase, and we guessed three wrong. And then we had 25 we predicted would not purchase, and seven of them did. So there's our our confusion matrix. At this point, if you were uh with your shareholders or a board meeting, um you would start to hear some snoozing if they were looking at the numbers and you say, "Hey, here's my confusion mat uh matrix." So let's go ahead and visualize the results. And we're going to pull from the map plot library colors import listed color map. Um, and this is actually my machine's going to throw an error because this is being um because of the way the setup is. I have a newer version on here than when they put together the demo. And we need our X set and our Y set, which is our X train and Y train. And then we'll create our X1, X2. And we'll put that into a grid. Uh, and we set our X set minimum stop and our X set max stop. And if you come all the way over here, we're going to step 0.01. This is going to give us a nice line, uh, is what that's doing. And then we're going to plot the contour, uh, plot the x limit, plot the y limit, and put the scatter plot in there. And let's go ahead and run this. Uh, to be honest, when I'm doing these graphs, there's so many different ways to do that. There's so many different ways to put this code together. to show you what we're doing. It's uh a lot easier to pull up the graph and then go back up and explain it. So, the first thing we want to note here when we're looking at the data is this is the training set. And so, we have those who didn't make a purchase. We've drawn a nice area for that that's defined by the naive bay setup. And then we have those who did make a purchase, the green. And you can see that some of the green dots fall into the red area and some of the red dots fall into the green. So even our training set isn't going to be 100%. Uh we couldn't do that. And so we're looking at our different data coming down. Uh we can kind of arrange our x1 x2 so we have a nice plot going on. And we're going to create the um contour. That's that nice line that's drawn down the middle on here with the red green. Um that's where that's what this is doing right here with the reshape and notice that we had to uh do the t if you remember from numpy um if you did the numpy module um you end up with pairs you know x uh x1 x2 x1 x2 next row and so forth you have to flip it so it's all one row you have all your x1's and all your x2s. Um so this what we're kind of looking for right here on this setup. Uh, and then the scatter plot is of course um your scattered data across there. We're just going through all the points that puts these nice little dots onto our setup on here. And we have our estimated salary and our age. And then of course the dots are did they make a purchase or not. And just a quick note, this is kind of funny. You can see up here where it says x y set equals uh xra y train, which seems kind of a little weird to do. Um, this is because this is probably originally a definition. Uh, so it's its own module that could be called over and over again. And which is really a good way to do it because the next thing we're going to want to do is do the exact same thing, but we're going to visualize the test set results. Uh, that way we can see what happened with our test group, our 25%. And you can see down here we have um the test set. Uh, and it if you look at the two graphs next to each other, this one obviously has um 75% of the data. So, it's going to show a lot more. This is only 25% of the data. You can see that there's a number that are kind of on the edge as to whether they could guess by age and income they're going to make a purchase or not. Uh, but that said, it still is pretty clear. It's pretty good as far as how much the estimate is and how good it does. Now, graphs are really effective for showing people what's going on, but you also need to have the numbers. And so, we're going to do from sklearn, we're going to import metrics, and then we're going to print our metrics classification report from the Y test and the Y predict. And you can see here we have precision uh precision of zeros is 90. There's our recall 96. We have an F1 score and a support. And we have our precision, the recall on getting it right. Uh, and then we can do our accuracy, the macro average, and the weighted average. Uh, so you can see it it pulls in pretty good as far as um how accurate it is. You could say it's going to be about 90% is going to guess correctly um that it that they're not going to purchase. And we had an 89% chance that they are going to purchase. Um, and then the other numbers as you get down have a little bit different meaning, but it's pretty straightforward on here. Here's our accuracy, and here's our micro average, and the weighted average, and everything else you might need. And if you forgot the exact definition of accuracy, it is the true positive, true negative over all of the different setups. Precision is your true positive over all positives, true and false. And recall is a true positive over true positive plus false negative. And we can just real quick flip back there so you can see those numbers on here. Uh here's our precision, here's our recall, and here's our accuracy on this. >> Welcome to this exciting journey into the world of statistics for data science. Have you ever wondered how data transforms from raw numbers into powerful insights that drive decisions? Well, statistics is the magic behind it all. Today, we will uncover how statistical methods help us summarize data, model uncertaintity, test hypothesis, and find relationships that can predict the future. So, buckle up. We are about to turn numbers into knowledge. Without any further ado, let's get started. Now, to start off, here's a key question. What are statistics in data science? Now, statistics is the science of collecting, analyzing and interpreting data. By applying statistical methods, we can uncover patterns in the data and make informed decisions. Now, as we continue, you notice how these essential concepts will form the backbone of many data science practices. Let's explore the key functions of statistics in data science. First, statistic helps summarize data using measures like mean, median and variance. Next, it models uncertaintity with probability and distributions. So, we can better understand risk and variability in our data. It also test hypothesis such as when we use AB testing to compare different outcomes. Statistics finds relationship through methods like regression and correlation revealing how variables impact each other. And finally, all these tools enable datadriven decision-m turning raw numbers into actionable insights. Now, let's talk about why does statistics matter in data science. Statistics form the backbone of data science, providing the mathematical framework needed to make sense of data and draw reliable conclusions. Without statistics, it would be impossible to turn raw data into meaningful insights or make confident evidence-based decisions. Now let's have a look at the main branches of statistics. Descriptive statistics and inferential statistic. So first let's talk about the definition. Then we have got methods and measures. Now in the case of descriptive statistics, now let's have a look at the main branches of statistics. So basically there are two core branches. Descriptive statistics and inferial statistics. Descriptive statistics summarizes and describes data using measures like mean, median, mode, range and standard deviation. Its purpose is to organize and present data typically with charts, graph or summary tables. The scope of descriptive statistics is limited to the sample data itself. Now on the other hand, inferential statistics makes inferences about populations based on samples. It uses methods like hypothesis testing, confidence intervals and regression. The purpose here is to draw conclusion and make predictions with common examples including AB testing and survey analysis. The scope of inferential statistics extends beyond the sample to the larger population. Understanding both these branches is essential for analyzing and interpreting data in any data science project. Now let's take a closer look at the descriptive statistics starting with measures of central tendency. The first measure here is mean which is the average of all the values simply calculated as the sum of all the data points divided by the total count. Next is the median which represents the middle value when the data is sorted from lowest to highest. This is especially useful when dealing with skewed distributions. And finally, the mode is the most frequently occurring value in the data set, helping us identify common patterns or repeated outcomes. Let's continue our deep dive into descriptive statistics by looking at the measures of variability. First, we've got the range. This is simply the difference between the maximum and minimum value in a data set showing us the spread of our data which measures the average of the square differences from the mean. This tells us how much the values in our data set differ from the average. Closely related is the standard deviation which is the square root of the variance. It gives us a more intuitive sense of how much the values typically deviate from the mean. And finally, we've got the interquartile range or we say IQR. This shows us the range of the middle 50% of our data, helping us understand how data is distributed across the center and avoid the effect of outliers. Understanding these four measures allows us to summarize not just the center of our data but how spread out and varied our data set is. Now let's look at some practical applications of descriptive statistics. One major use is the data exploration and summarization where we quickly get an overview and basic understanding of complex data sets helping track performance and detect problems early in fields like manufacturing or operations. And finally, they are central to business reporting and dashboards where concise summaries are essential for managers to review trends and make datadriven decisions. So in short, descriptive statistics help transform raw data into clear, actionable information across many business, scientific and operational context. Let's explore the first type of data in statistics, qualitative or categorical data. This type of data includes descriptive information that cannot be measured numerically such as categories or labels and yes or no responses. These are all about qualities or characteristics rather than quantities. Qualitative data can be further divided into two types. Nominal where the categories have no specific order and ordinal where the categories do have an order or ranking. Recognizing and classifying qualitative data is very important as it affects how information is analyzed and interpreted in statistics. Now let's discuss the second main type of data in statistics which is quantitative or numerical data. This type of data includes information that can be measured and expressed with numbers making it ideal for mathematical analysis. Quantitative data is further divided into two main categories. First there is discrete data. These are countable values with specific fixed points such as the number of students, cars sold or website clicks. Second, there is continuous data which includes infinite possible values within a given range. Examples of continuous data include height, weight, temperature or time. Understanding the distinction between discrete and continuous data is very important as it determines which statistical methods and visualizations will be most appropriate. Now let's dive into the fundamentals of probability. Probability measures the likelihood of an event occurring and it's always expressed as a value between 0 and 1. Here are some key concepts. If the probability or P equals to zero, that means the event will never occur. If P is equals to 1, the event will always occur. And if P is equals to 0.5, the event has an equal chance of occurring or not occurring. It's truly a 50/50% scenario. Now, understanding these basic principle help us quantify uncertaintity and make informed predictions about future outcomes. Now let's look at the different types of probability. First there's classical probability. This is based on equally likely outcomes such as flipping a fair coin or rolling a balanced die. Next empirical probability which relies on observed frequency. It's calculated from actual data such as the proportion of rainy days over the past month. Let's say for example the probability of it's raining today given that it's cloudy. Understanding these three type help us choose the right approach for different situations. Whether we are predicting outcomes, analyzing data or making decisions under uncertaintity. Now probability has a wide range of powerful applications in data science. First of all, it is used in predictive modeling and machine learning where algorithms estimate future outcomes based on existing data. Probability also plays a key role in risk management and decision making helping businesses and researchers evaluate the likelihood of different scenarios and plan accordingly. Conditional probability calculating the chance of one event given that the another has occurred. This is crucial in fields like healthcare, fraud detection and marketing analytics. And finally, probability is foundational in AB testing and experimental design, allowing us to measure the effectiveness of new strategies or products. These applications show how probability enables smarter evidence-driven progress in modern data science. Let's look at one of the most common probability distributions. The normal distribution. This distribution is famous for its bell-shaped symmetric curve which shows that most values cluster around the mean with fewer and fewer values appearing as you move away from the center. Now many natural phenomena like heights, test scores, and measurement errors tend to follow this pattern making the normal distribution a key concept in statistics. It's characterized by two main parameters. The mean which determines the center of the curve and the standard deviation which controls its spread. Recognizing the distribution help analysts make predictions, calculate probabilities and apply statistical techniques to real world data. Next, let's explore the binomial distribution which is another common probability distribution. The binomial distribution is discrete and is used for situations with binary outcomes like success or failure. It is based on a fixed number of trials where each trial has a constant probability of success such as flipping a coin a certain number of times or tracking pass fail rates. Examples of binomial experiment includes coin flips and measuring how many students pass or fail a test. This distribution help us model and analyze outcomes when only two possibilities exist in each trial. Now let's focus on the poion distribution. Another key type of probability distribution. The portion distribution is a discrete distribution specifically used to model rare events. It's particularly helpful for modeling events that occur independently over a fixed interval of time or space. For example, it predicts how many times an event like a customer arriving on a website, receiving a visit might happen in a certain period. Typical examples include customer arrivals at a store, defect rates in manufacturing or counts of website visits over a set period. This distribution is great tool for understanding and predicting random independent events that don't happen very often but are important to track. It's a branch that allows us to make inferences, predictions, or generalizations about a larger population using sample data. Here are some key concepts. First is the population versus sample. The population represents the entire group we want to know about while the sample is the subset we actually collect data from. Next concept is sampling distribution which refers to the distribution of a statistic across multiple samples from the same population. Standard error measures how much the sample statistic is expected to vary due to random sampling. And finally, margin of error tells us how much we can expect our estimates to differ from the true population value. Together, these concept form the foundation for drawing reliable insights from sample data in inferential statistics. Let's review some of the main techniques used in inferial statistic. First, we've got hypothesis testing. This method allows us to test claims or ideas about population parameters based on sample data helping us determine if observed results are statistically significant and reliability of our estimate. Next are confidence intervals. These provide a range of likely values for a population parameter giving us a sense of possible variation reliability of our estimate. And lastly, regression analysis is used to model and analyze the relationships between variables, allowing us to make predictions, uncover trends, and understand how changes in one factor might affect another. Now, let's walk through the hypothesis testing process. The first step is to formulate hypothesis. Start with null hypothesis represented as Hnot, which states that there is no effect or difference. Then there's the alternative hypothesis represented as H1 which suggests that an effect or difference does exist. Now after setting up the hypothesis the next step is to choose a significance level noted by alpha. Common choices for significance levels include 0.05 5% 01 1% 010 which is 10%. The significance level is important because it controls the likelihood of making a type one error also known as false positive. And now each step in the process is critical for ensuring that statistical results are both meaningful and reliable. The next step is to collect and analyze sample data. Ensure the data is represented by choosing a good sample. Then calculate the appropriate and test statistic for your hypothesis test. Once that's done, it's time to make a decision. Compare the p value to the chosen significance level alpha. Now, if the p value is less than or equals to alpha, you reject the null hypothesis. If the p value is greater than alpha, you fail to reject the null hypothesis. And finally, interpret your results. Draw your conclusions with respect to the context and problem at hand. Always keeping the bigger picture in mind. Following these step helps ensure your hypothesis test is robust, clear and meaningful. Let's review the common types of hypothesis test used in statistic. The one sample test compares the mean of a sample to a known value and the one sample zed test is used when the population standard deviation is known. Next, we have two sample test. The independent samples t test compares to the means of two different groups while paired samples t test compares before and after measurements for the same subject. For categorical data test, the shear test checks for the independence or goodness of fit. And fiser's exact test is useful for small sample sizes. And lastly, non-parametric tests such as the man Whitney U test and Wil Coxson signed rank test serve as alternatives to the t test when data doesn't meet certain parametric assumptions. Choosing the right hypothesis test depends on your data type and the specific question you want to answer. Let's explore the central limit theorem or CLT of foundation for inferial statistics. The central limit theorem states that as the sample size increases, the distribution of sample means approaches a normal distribution even if the original data is a normally distributed. This sample works for any population distribution making it incredibly powerful. Now for good results, the sample size should typically be 30 or more. What's interesting is the sample mean distribution which will have the same mean as the population. The standard error which measures variability by the sample mean equals sigma / the square roo<unk> of