Deep Learning Full Course 2026 [FREE] | Deep Learning Tutorial | Deep Learning Course | Simplilearn

Key Takeaways

understand patterns, make prediction, and solve real-world problems almost like a human brain. That is the power of deep learning. Hello everyone and welcome back to Simply Learn. In this video, we will explore the complete beginner to practical journey of deep learning. We will understand how neural networks work, how machines learn from the data, and how frameworks like TensorFlow and PyTorch help us build real deep learning models. This topic is important because deep learning is behind many of today's most powerful AI applications, including image recognitions, speech assistants, recommendation systems, natural language processing, self-driving cars, and even generative AI. If you want to build a strong foundation in AI and machine learning, deep learning is one of the most important skills to learn. In this session, we will cover the basics of deep learning, artificial neurons, perceptrons, weights, activation function, forward progression, loss function, gradient descent, and backpropagation. We will also move into practical implementation using TensorFlow, Keras, and PyTorch. You will see how neural networks are built, trained, evaluated, and applied to real classification problems using data sets like fashion and a Keystone-style projects. By the end of this video, you will understand how deep learning models learn, how training works, and how to start building your own neural networks using modern deep learning frameworks. Before we move ahead, let me quickly share you something exciting for anyone who wants to build a strong career in generative AI and machine learning. The E&ICT Academy IIT Kanpur Professional Certificate Course in Generative AI and Machine Learning is designed to help you learn the complete AI and ML skill set from Python, data science, machine learning, deep learning, and NLP to generative AI, prompt engineering, LLMs, computer vision, and reinforcement learning. The program includes live online classes, masterclasses designed by IIT Kanpur faculty, and practical exposure to popular tools like ChatGPT, Hugging Face, DALL-E, TensorFlow, Keras, Gradio, LangChain, OpenAI, and more. You'll also work on 15 plus hands-on projects covering real-world use cases like employee attrition prediction, loan default analysis, AI-powered HR assistance, text-to-design platform, recommendation engines, sales forecasting, and deep learning applications. On completing the program, you'll receive a program completion certificate from E & ICT Academy IIT Kanpur along with official Microsoft course badges for eligible learning paths. So, if you want to go beyond just using AI tools and actually understand how generative AI and machine learning solutions are built and applied in real-world business problems, this program is worth checking out. The link is given in the description box below and in the pinned comments. Go check it out. Before we continue, here's a quick question for you to answer. Which of the following is commonly used as an activation function in hidden layers of neural networks? Is it ReLU, HTML, SQL, or Excel? Let me know your answers in the comment section below. Let's get started. >> I I've always found that in in teaching this several times uh in this program, getting to the deep learning is um challenging. Doing the deep learning is challenging, but very rewarding and I think a lot of people enjoy the topics here cuz they're very um modern and very relevant to um very advanced kind of modeling uh techniques. So, a lot of people enjoy it, but it is going to be a little bit more challenging than what we've done so far. So, but it's okay. I think we can handle it. Um So, yeah, let's start with our introduction here. Okay. So, what we're hoping to cover here are just the uh some of the background mostly just background information about deep learning. So, some of the um achievements that have really happened in recent times. Um and we're talking, you know, less than 10 years old and some of these achievements that are uh have brought deep learning to the forefront and made it a really popular uh field. So, we'll understand some of the applications and also some of the challenges of deep learning. It's not Deep learning is not perfect. It has some drawbacks and some challenges that make it um difficult at times. So, we'll talk about that. And then we'll talk about some of the frameworks. Kind of already mentioned uh PyTorch and TensorFlow as the two main ones. Just talk about that a little bit. Um and then talk about what are the main steps of any deep learning project that uh we would want to follow. I think what you're going to see there is they mostly mimic the same kind of steps we were doing in machine learning. Things like uh data preparation, then kind of building a model, training it, um evaluating it on test data. Those kind of steps are mostly the same. It's just the details of those are going to be drastically different. You know, that training process is going to be more involved, I think, with with deep learning. Um building the model is a little more hands-on with deep learning as well cuz you're literally going to be architecting and constructing a neural network. It's much different than just instantiating a um a model from scikit-learn, which is pretty easy to do. Um there's going to be more hands-on work involved in in building these deep learning uh networks that are going to be our models. Um so, we'll talk about that process. Okay? All right. So, let's start with just AI in general. So, the reason to start here is just to see where deep learning kind of fits into the rest of the stuff that we've studied so far. So, just machine learning. And um historically uh where has deep learning fit in? So, I think we've seen this picture before um early on in our machine learning and even data science uh uh courses. But here it is again. Um where we have AI as this pretty broad umbrella um that is um basically involving machines, so computers, um trying to perform tasks that normally humans would have to do. And what we just got finished studying was machine learning. Right? So, machine learning is primarily having the machine learn from data how to make a decision, right? How to complete some task, whether that's prediction or maybe like an unsupervised case, um doing things like compression, dimensionality reduction, or finding patterns and clusters. It's learning from data how to do that, right? That's machine learning broadly. Um where deep learning fits in is really a subset of machine learning because we're still learning from data. Uh we still are doing that. It's just we're using a specific tool to do that. We're just going to be a neural network. Um so, that's why in this diagram you see deep learning as kind of a circle embedded inside of the machine learning circle because it's it is kind of a sub uh field of machine learning. You're still learning from data. You're just doing it in a very particular way. Um so, it's kind of like a specialization. That's why I said earlier, not everybody studies deep learning because you can get pretty far without deep learning. You still have this whole field of machine learning and you can you can still do a lot with that. Um but this is certainly kind of a specialization is drilling down into working with neural nets in particular as your model. It turns out that working with neural nets is really powerful for many reasons. Um and what we're going to see is that deep learning really unlocks uh working with specific types of data that we have not worked with so far. And you know, we already have a sense of that, things like images, things like text, we really haven't worked with too much or at all. Um but deep learning really unlocks the ability to do that. Um so you can see that here. Yeah, there there are um we've briefly talked about a couple of them. Uh so inside of machine learning that is um not necessarily deep learning would be something like RL. So you So it it's not really a small circle, but probably another big one is uh and it it honestly kind of overlaps with deep learning because you can use uh neural networks inside of reinforcement learning. So RL is short for reinforcement learning. I draw it overlap just to say that nowadays a lot of RL is done with deep learning ideas in terms of using neural nets, but it wasn't always that way. But yeah, there's RL and then there's uh certainly like a lot of use cases of unsupervised learning inside of machine learning that does not use neural nets. So unsupervised learning So things like clustering uh unsupervised learning. You know, thing basically things that do not use neural nets, but are but are still considered machine learning cuz you're still learning from data. Yeah, basically everything we covered in the last course. Um so su even supervised learning that's not using neural nets is still in this in this bucket of machine learning. All right. So So this is this is where So we're studying deep learning we're going to be in this course. This is where it kind of fits in. It's just a subset of machine learning cuz at the end of day we're still learning from data. It's just going to be using a particular type of model, which is a neural net. And more so than just a basic neural net, it's often going to be what we will um term as a deep neural net, meaning it will be a pretty complex model that has uh a lot of layers to it. Um as we will see, but it is just a subset of machine learning. Okay. So, historically, how is this evolved to get to that point? Um well, we had original the the very first model of a neuron was proposed in 1943 um by McCulloch and Pitts. So, they produced a paper kind of um proposing a potential model of the of the brain and the neurons in the brain uh in 1943. So, obviously, a long time ago. And at the same time, some other things going on. We had the Turing test, which you may have heard of, basically um testing the intelligent ability of a machine. And so, we had that going on as well. Obviously, other things going on from the '40s to '50, but the big one that's relevant to deep learning is the very first proposed model of the neuron. That was as old as 1943. It was more of an idea then. Obviously, they didn't have computing power to do anything with that. Um it was more just the idea. The concept was kind of as old as that of just modeling an actual neuron. Um it wasn't until the '50s that we see the actual um term artificial intelligence being adopted um as well as the very first very basic neuro uh neural network model, the perceptron, uh that was invented in 1957. At the same time, we have some programming languages being invented uh to take advantage of machines. Um not Python yet, obviously, but you know, some older languages that end up inspiring the uh generation of Python later on, but still some some uh high-level languages that are invented. But, this perceptron is really relevant. Actually, we're going to study perceptron in the next lesson as kind of the first model, the first basic neural network model that will propel, you know, deeper and deeper networks built off of the idea of kind of a perceptron. So, we'll talk about perceptron quite a bit. Okay, so then we go to the '60s. Are perceptrons still being used today? Perceptron itself is probably not used too much, but the idea of the perceptron is very present in so many network. Think of it like the building block of any neural network was founded on the perceptron idea. So, the the perceptron itself, like just a single neuron. So, so we're going to learn about this, but the perceptron is basically the model of a single neuron. A single neuron not quite used too much. It's very basic. But, it forms the building block for nearly all neural networks that are around today. So, I would say, yeah. I would say it's it's inspired pretty much any neural network is inspired by the perceptron. So, it's a very, very important idea and concept, and we're going to spend some time talking about it and studying it as kind of our foundation for neural nets, and then we'll build on that by kind of stacking stacking multiple multiple perceptrons essentially together until we get a larger and larger network. Yeah, so the thing is the idea's been around a long time, but there hasn't been the, you know, computing power to really take advantage of it until recently. And when I say recently, I mean probably last 15-20 years. Hasn't been Right, there hasn't been the computing power available to really unlock the capabilities of these and make make uh and larger networks that can solve harder and harder problems. So, we don't see that uh we don't see that until, you know, recent. First, I was a keypunch operator. I think it's the uh don't know what keypunch is. I think it's the um like the binary uh the bit uh cards that uh fed the original kind of computers or older computers. And you had to literally punch out the bits. Also, this time in the '60s was the original chatbot, Eliza, um 1966. Now, it's nothing like generative AI, right? This is This chatbot is all rules-driven. So, nothing data-driven, nothing like that, certainly no neural net, nothing that we see today with these transformer-based models and GPTs and such. Th- But, it is a chatbot in the sense that you can input text and then it can respond with text. Basically, it's all rule-driven though, so it had to be pre-programmed uh how to respond to things. But, that was, you know, the first kind of AI chatbot. Okay, so then we see kind of a uh lull in the uh '70s to '80s 70 to 80 um interest in AI drops significantly. Um funding from from government sources was insufficient. So, there's kind of a lull period here. Now, that's not to say that um there wasn't anything going on. You know, there's still statistical research going on and and advancements in the machine learning community, but nothing really relevant to neural networks uh at this period. But, still, you know, some of the some of the developments in the in the statistics community are still relevant during this time uh to machine learning. Just not necessarily deep learning. But, there's a rebound of that in the '80s. Um and you may be able to guess why, maybe due to geopolitical things going on in the '80s, like the Cold War. Um we see a revival in uh research and dedication to AI. Um we also start to see some communities uh develop um that dedicated to the advancement of AI. So, the Stanford host the inaugural uh conference, the AAI, um which is kind of the uh inspiration for some of the conferences we see today, like NeurIPS and and things like that. So, in the '80s, there's kind of a revival. And then there's a kind of a um interesting period of time in the late uh late '90s. So, kind of early, I would say even early '90s to early 2000s. In this period, um we see a significant investment in AI from the big players, like like Facebooks and Netflixes of the world, and Google, of course, Twitter. They start investing in AI quite a bit. 1997, we have the IBM Deep Blue, if you've ever heard of that, um beat the uh world chess champion. Um so, that was a really cool kind of AI achievement. But, this is this period's called out here in particular because AI it's particularly deep learning had not advanced that far even by this point. However, there are some significant investments in this period that lead us to kind of where we are today. Um I would say two of them that are not necessarily listed here, but will be coming up, are where it says companies like Facebook, Twitter, and Netflix, and Google not mentioned, but should be there. They really start investing in AI. I want to call out Facebook and Google in particular because they build the frameworks that we use today. So, Facebook invented PyTorch around this time. And Google invented TensorFlow around this time. They did those independently. Now they are mostly open-sourced, so anybody can use them, but they were in-house tools to work with neural nets. Everybody wanted to work with neural nets. They had their own in-house tools like PyTorch and TensorFlow to kind of build and manage neural nets at the time. And this all started right around this period, 2006 or so. You know, not that long ago relative to uh history. Only about 20 years. Not that long ago. And we see kind of those foundational libraries being developed. And obviously there's been a lot of advancements since then. So, in uh the period after that, um this is kind of where the deep learning golden age is. Like really the 2010s onward. 2012 is called out here. We're going to talk about there's a really big event in 2012 that happens, which is the AlexNet convolutional net is is uh is published and studied. It's a really big computer vision model. So, a lot of really amazing achievements from this period, really 2010s onward in deep learning. And it kind of coincides with hardware being more available, um neural nets being more readily available with those frameworks being more mature and developed like PyTorch and TensorFlow. Um more people have access to them. They're open-sourced, so more people can work with them and and really get their hands on it, including the research community. So, there's a reason it kind of blows up in the 2010s onward and even today. Uh it's So, I'll tell you why. It's because, um, a lot of generative AI is powered by the transformer. And the transformer was not developed really until 2017. So, that's less than 10 years ago. Um, but the really the computing power and data availability to work with something like a transformer has not caught up until really like the last five or so years. So, that's why it's so recent. The model that they're all built off of is less than 10 years old. Um, the transformer. But, it's more te- like just other things like hardware and data availability. We've caught up to that. Uh, and and now we can have these really massive language models that power generative AI because compute power is relatively easy to to manage and, um, data's pretty easy to get your hands on, too. But, even But, that's what I'm saying. The model is not that old that powers it. All right. But, really good question. Um, so, what are some of our motivations for deep learning? So, there's a number of topics that we're going to study. You know, they're all listed here. We're going to start with the perceptron. We're going to to talk about these topics later on that have to do with like, uh, the back propagation, really having to deal with how to train a neural net. Um, so, we'll talk about how how they get trained. We'll talk about the earliest model of a neuron, which is going to be the perceptron. So, we'll certainly cover that. Um, then we can stack perceptrons together in what are what is known as a multi-layer perceptron. That's kind of the next level. Um, that's kind of like a miniature deep neural net. It's It's not very deep, but it does have multiple layers to it. Um, and then we have the kind of in more recent times these very advanced networks like these that we're going to study um convolutions mainly being used for images and recurrent neural networks mainly being used for text. So, we're talking about the motivation. One of the motivations one of the primary motivations for talking about deep learning is really this. This idea that deep learning is going to unlock our ability to work with text data and our ability to work with image data. There are so many problems out there that work with this kind of data which in the community is known as unstructured data, right? So, so far we've really only done problems with structured data. And structured meaning that it's basically a matrix that has rows and columns with really well-defined features, right? That are rows and columns like our like our housing data that has square footage and bedrooms and bathrooms. That's all very well structured. Text is not structured at all. It can be variable length. You can have paragraphs. You can have whole documents. It's unstructured. Um images, same thing. They can be many different resolutions. They have a little bit more structure to them in the sense that they are matrices somewhat, right? They have um rows and columns of pixels, but still they're considered unstructured data. Um and and uh so, neural nets and deep learning will unlock the ability to work with those. There's so many problems that work with images like object detection, image recognition, you know, and that extends to so many applications. And then working with text obviously, generative AI working with text or even doing things like sentiment analysis or language translation. A lot of things work with text. So, we need these advanced neural nets to be able to deal with that kind of data. The The models that we've studied so far just are not capable of working with that data very effectively at all. Like logistic regressions or decision trees or even things like an XG boost, they're going to struggle on that unstructured data. They're not really going to be able to process it. Neural net on the other hand is going to be able to. That's one of our motivations for studying it is it unlocks this whole set of applications for us that we couldn't do otherwise. Namely working with text and image data, right? Okay. So some particular tasks there that we were that we are going to study. So things like image recognition. Um now the very basic example that I probably have said before, but you know, something like being able to predict if an image is a cat or a dog or if an image is um a cat, dog, a giraffe or a a wolf or whatever it is, right? So image recognition tasks and that really extends into so many applications like object detection. So working with images, really huge deal. We're going to devote a lot of time to it, um but we really need neural nets to be able to do that effectively as we are going to find out. Um things with language like understanding and processing human language. We have these transformers that do this so effectively, but it wasn't always that way and it was nearly impossible to do um with basic uh models. Just nearly impossible to understand language at the level that we see today. If you think about the the amount of language understanding it takes to have these um things like GPT and Claude and stuff, it's just immense. And they're but they're really huge neural networks that are doing that and they're learning against so much data. But we really need deep learning for that. Um and other applications, things like speech recognition. You know, we haven't talked about it, but another type of unstructured data is um, audio. Right? So, audio data, being able to do things like recognition of audio, translation into text. That's a really hard problem, basically impossible without neural nets. Uh, to be able to recognize speech in audio. Same thing with video. I know it's not listed here, but video certainly unstructured. We need uh, neural nets to be able to process those um, as well. So, basically any type of unstructured data, we're going to use neural nets to be able to process them. Yeah, a lot of a lot of uh, neural nets. So, speech recognition is powered by We're going to cover this later with much later on in this course when we get into RNNs. Um, a lot of things like um, Alexa devices, for instance, have been powered by RNN neural networks to uh, basically recognize your commands, your voice commands. That's all neural network powered. Right? To be able to recognize a voice command, translate that into some text, and then use that text for creating uh, tasks or doing things or or generating new text to be able to respond. Yeah, they're That is, yeah, they're neural nets. So, this speech recognition part is all um, neural net driven, namely RNNs, recurrent neural nets. Um, and and in modern times, RNNs {slash} transformers, we see transformers kind of taking over this space as well. A lot of language stuff is being taken over by transformers. But traditionally, they've been RNNs. Okay. So, um, let me give you some breakthroughs. So, for example, you know, we talked about this period of 2010s, one of the uh breakthroughs in deep learning was in 2012, the AlexNet paper, which I encourage you to go look up and and read a little bit about. So So, the AlexNet is a type of convolutional neural net, and it introduced really the convolution as a preeminent operation on image data to be able to classify images into different categories. Um so it it it this network architecture using convolutions um blew away the competition in the ImageNet Challenge, which was a uh basically taking a bunch of images. So, you can see here like the the training data has 1.2 million images. Testing data has more has 1.5 million images. Your goal is to categorize those images into 1,000 different categories. So, So, think of it as like 1,000 different animals. Not just cat or dog, but 1,000 different uh animals. And um the best methods at the time um mhm things like basic neural nets or maybe even like an extra boost were blown away by this convolutional net um by more than 10%, which is astronomical amount in in research um of this uh at this kind of level. So, this was a huge breakthrough, and what we see from there on is convolutions becoming kind of the default building block of anything working with images. So, like object detection, it's all going to use convolutions, and it was all inspired by this AlexNet paper. And it was for the first time really like people realizing like, "Hey, these neural nets can really work on unstructured data very effectively." Because the methodologies to work on these images prior to this with was to basically try to generate a bunch of structured features, which was not very effective. Going from like an image into a flattened vector and trying to generate features out of that to pass through like an XG boost was the strategy, but it got surpassed immensely by this convolutional net. And then we see just an explosion of computer vision progress with people using uh people using uh convolutions. So, one of the things we're going to study in this course is convolutions. They're really important for computer vision. If you're ever going to do anything with images, uh any kind of neural nets processing images is going to use convolutions in some fashion, right? In the architecture. So, we'll devote some time to studying convolutions for sure. But this was a huge breakthrough at the time. And it again, 2012. That's really not that long ago. It's It seems It's hard to believe that this was 2012 um and you know, that is kind of a long time somewhat, but really not really not really in historical terms. Yeah, please check those out. I think the AlexNet is really fascinating. Now, it when you read that stuff, it may not make any sense to you right now. That's okay. It should make a lot more sense by the time we get into uh computer vision and start studying convolutions. I think it'll make a lot more sense. So, maybe hold on to those. I encourage you to read them, but they may not make any sense right now because we haven't covered anything about neural nets really um or especially convolutions. So, they probably won't make any sense to you, but um I promise they will. They'll make a lot more sense by the time we go through and talk about convolutions. Okay. So, you know, we'll talk about some more breakthroughs. That's just one of them. Uh however, there are challenges uh that were encountered along the way. Obviously, like a lack of research funding was a big challenge. Probably bigger like bigger than that definitely are these next two which are data and hardware. These are the two things that stalled deep learning progress for a long time. Hardware mostly because neural nets and their structure is something we're going to learn about benefit immensely from advanced GPU hardware. It's primarily because the computations that a neural net does benefit a lot from the parallel parallelization that a GPU can provide at the hardware level. And so only until recently have we had a lot of advancements in hardware such that GPUs are readily available. Wasn't always that case and also wasn't always the case that that the code the Python code could take advantage of a GPU. But it it can today a pretty easily with our frameworks like PyTorch and TensorFlow. They they make it easy to take advantage of a GPU hardware. And so that's why we've seen an explosion in the last 10 15 years because GPUs are more readily available and can be taken advantage of by the code. Same thing with data. Data is more readily available, right? So there's a lot more of it a lot more it's it's cheaper to store it a lot easier to collect it. So it wasn't always that case and what the case is with neural nets is you need a lot of data to train a neural net more than what we've used so far for the structured data problems that we've dealt with. So the the the expensive nature of storing data and getting access to it was also prohibitive to a lot of progress in deep learning. But that's that's been kind of fixed, right? It's now a lot cheaper than ever and consequently we've seen a big explosion, right? Last 10, 15 years. Okay. All right. Any questions uh so far? Just some background, right? Just some background and little historical context. Um haven't gotten to too much into it yet, but just again, this is the point of this first lesson, just more background information, historical context. And then we'll start studying the models of the neuron and working our way up to, you know, more complex neural nets. Okay. Let's talk about deep learning then uh a little more in particular. So, we just covered the fact that um deep learning is really a subset of machine learning focusing on using neural nets, and often what are going to be deep neural nets, which are going to be neural nets with uh a lot of layers, a lot of um complex structure to it um to be able to uh learn from data. So, learning from data is machine learning, right? But using neural nets to do that is deep learning. So, that's where we're at is using neural nets in particular, moving away from just basic machine learning into more specialization here with neural nets. So, um what's nice about neural nets is that it can work with structured data, but as I've said, it can unlock the ability to work with unstructured data from many different domains such as imagery, audio, video, text um to make predictions uh usually. Um and the uh this has unlocked a lot of progress in domains like computer vision, so object detection, self-driving cars, those kind of things. NLP, which is dealing with language, so language tasks like sentiment analysis, translation, text generation. Um even working with audio data like speech recognition, so a lot of progress in those fields. And deep learning is going to be able to surpass a lot of the machine learning techniques that we've seen because they have a very uncanny ability to learn complex patterns. Um one of the things that is that gives that the ability is how complex the model is. But the other thing is it's going to use a lot of data. So it's going to have a lot of examples to go off of. It needs that in order to extract those complex patterns amongst imagery, text, audio, video, right? It's going to require a lot of data. So that's something we'll see when we start doing examples. Okay, so just to compare the two. So we're talking about deep learning and machine learning and remember deep learning really being a subset of machine learning. Machine learning is a little bit more broad um where we have, you know, unsupervised learning, supervised. A lot of the problems in deep learning are still supervised. Meaning that they're still a label. Like if you have an image, there's still a label of what that image is. Is it a cat? Is it a dog? Is it a giraffe? Right? Or one of those thousand categories like an ImageNet uh challenge. So there's still a supervised nature to a lot of these deep learning problems. It's just the model that we're using is now a deep neural net, right? Instead of a decision tree, instead of a random forest. And also, the data is inherently different, right? It's going to be imagery. It's going to be text, potentially audio, video. So we're moving away from structured data and really focusing on unstructured use cases with things like images and text. One of the things that's true that we've seen so far is especially with machine learning course we just finished, we saw that it's really important to get those features right. You know, we have to do some data prep to get our features correct, get rid of any null values, maybe even engineer some features like scaling it or adding two features together. Um it's really important to do that. What we're going to see with deep learning is it's not that important to do that. It's not that important to do feature engineering at all because the the network's going to be so powerful at picking up on patterns from the original data like the original images. Um we're not going to need to do a lot of manual feature engineering with with deep learning. So, that's going to be one kind of advantage here is not a lot of manual uh feature engineering. And that's actually the the ImageNet uh challenge we just talked about with AlexNet uh convolutional net winning that challenge, that was the thing everybody was doing was trying to do a lot of manual feature engineering so that they could use like a a classical machine learning model like a XGBoost. But, doing that wasn't as effective as just a really good neural net that can just extract patterns from the raw data very effectively using something like convolution. Okay, so deep learning, as I've said, really excels in those kind of tasks like image recognition, NLP, speech uh recognition. Um so, we've talked about that. Uh it NLP, remember, is short for natural language processing. So, you'll probably hear me use that word NLP especially when we get to later on and we're studying things like transformers and RNNs. Those are just language tasks, right? So, think of text working with text, text generation, sentiment analysis, uh speech or language translation, I should say. Um so, processing natural language, which is really text, right? For for our use cases. Deep learning often requires really large data sets, so much more data than you see in a machine learning technique. Um and also more computational resources. So, this is think GPU. Most deep learning is going to benefit from using a GPU, whereas machine learning doesn't really use that at all. Like if you do a decision tree, an SVM, a logistic regression, you're not using a GPU at all. You don't need to. You it will still be pretty effective um without that. And uh whereas deep learning a lot of models will take forever. Just to give you some context, like the um GPTs of the world, these transformers that are, you know, neural network based, they train on data that's about the size, if you took all of the text from Wikipedia and more. So, all the text from Wikipedia um plus maybe all the text that would be in an encyclopedia collection. Think of a data set that big as a prototypical training data set for something like a GPT. And think of thousands of GPUs working together. Even in that case, it can still take days to train a model. Days. Not minutes, not hours, days, because the models are so big and there's so much data being processed to train it, that is how much time it takes to train those kind of models. So, there's substantial amount of resources. Um which is why there's so many uh news articles written about, you know, Nvidia and partnering with these uh generative AI companies like OpenAI and Microsoft, etc. So, a lot of resources are are needed. Now, for the networks we're going to build in the in the program, not so much. We'll we'll still explore using a GPU, and we're actually going to practice using a GPU. Um we will have access to free GPUs in Colab. So, that's really nice. So, even the Simply Learn lab environment has free access GPUs. Um, so we will practice using it and it will benefit us for things like image recognition, working with imagery or RNNs or transformers even. Um, they'll benefit us, but just as a rule of thumb, deep learning requires more data, more resources to to train. Computational resources. Well, so if you if you have a Windows machine, sure. Um, you can use the Nvidia GPUs. If you have a Mac, um, like a MacBook, um, even like one of the M1, M2, M3, M4 series, those have built-in GPU capabilities that, uh, we can take advantage of. So, I'll share those resources with us when we get to building our models, but even if you have like one of those M1, M2, M3, M4 Macs with the the silicon chips, they can use their internal GPU to train models. Um, and I've done that before and it's really nice. It speeds up the training a lot. We can also use Colab. So, Colab offers free GPU access, uh, which is nice. Um, so we can use that, too. And we will. So, we'll practice that as when we get to it. Okay. And then finally, deep learning, you know, we're going to be using neural nets that are potentially very deep in the sense that they have a lot of layers. So, we haven't studied what a layer is yet. That's okay. Just think of a really complex model with a lot going on with it. That's going to be, you know, way more complex than something like a decision tree or SVM or random forest even. So, that's what we're talking about here. To train deep learning, we, uh, often will use GPUs, uh, because they're going to be really effective. Most traditional machine learning methods that are not deep learning can get away with just using a CPU because they have very simple algorithms that are that are training that do not require using the underlying GPU resource that can really speed up deep learning. Okay? All right. So, I wanted to take us through some of the successes that have been recent. Some of the foundational kind of moments. And there's been growth in a lot of areas. So, working with audio, computer vision, even reinforcement learning that has been powered by deep learning. Most reinforcement learning algorithms prior to deep learning were based on what are called kind of tabular methods. Where they were keeping track of rewards and and data in in a more structured format. Uh But deep learning has since made a lot of progress in simplifying that and and making it much more powerful and even in reinforcement learning. We'll talk about one of those cool moments there. So, back to the AlexNet, you know, we talked about this in 2012. This was the champion in the renowned ImageNet challenge. This is where again they used convolutions to really power learning imagery and being able to classify images. And this this was so groundbreaking that a lot of moments in deep learning over the years have been referred to as kind of a ImageNet moment or AlexNet moment. Because [clears throat] how big of a breakthrough this really was. In 2012 this convolutional model was kind of created. Now, we'll study this architecture later on if what this actually means, what these uh layers are doing, what is a convolution. Um we'll study all of that when we get into computer vision, but this was a huge breakthrough in working with image data. Then in 2013, we have a really important breakthrough of working with text known as uh the first kind of word embedding model, which is really really important for where we are today because word embeddings play a really critical role in working with text uh because these are um models that map text into numerical vectors, which is really critical for um working with language modeling. And every transformer really has a word embedding at the very front of it to take raw text or tokens and essentially um make those into a mathematical vector so the neural net can work with it. The very first uh model there that was a big breakthrough, this is 2013, um was a model called word2vec, uh which uh you may have heard of. This was uh used for uh preprocessing text essentially to to map it from raw word or tokens into uh vectors. So, really really uh critical for any kind of language model to do this. And so, this this has been around for a little bit longer than the language models, but this was a really important breakthrough for them for it for generative AI. So, every generative AI like GPT, Claude, every everybody has a word embedding model that's at the very front of their uh transformer-based uh text generation model. So, this is this is a huge breakthrough. Then we have the very first kind of uh uh, language models known as sequence-to-sequence. Now, this is not using a transformer. So, this is using RNNs to try to do NLP tasks like a language translation from uh, German to English or French to English or or English to French, whatever it is. And they tried to to make NLP, so doing things like translation, language generation, sentiment analysis, they tried to make it a lot better and it did. So, at the time, RNNs were kind of the best language models around and this is 2014, but they quickly get surpassed by transformers not too long after that. And that's where we are today. Everything is kind of transformer-based, but at the time, RNNs were kind of the best uh, for language. We will study RNNs uh, later on in this course. Recurrent neural nets. RNNs, which are just a flavor of deep learning that uses uh, a special architecture. So, here it is again, just some different variations. There's a lot of different variations on the RNN uh, architecture um, that attempts to predict um, again, what word should be next in the sequence using the RNN. We do that very well today with transformers rather than RNNs, but at the time, these were the best around. Okay, right around the same time in the image generation uh, field, we have GANs as a big breakthrough. So, what what are GANs? They are uh, generative adversarial networks. Um, these are models that generate images, fake images um, and they do it through a competing uh, adversarial networks that um, one is a generator that builds images, the other is a discriminator that's supposed to predict if the image is real or fake and they kind of boost each other up so that the generator, by the time the training is done, is so effective at generating fake images, they kind of mirror real images. This actually led to the explosion in deep fakes. So, deep fakes became a real issue. They're still issues today, but the the first issue the first time it was an issue was really when GANs were prominent. They were developed in 2014. So, again, not too long ago. Most models that generate images today are actually not GANs. They're what are known as diffusion. So, like Dolly and and the Google banana models that you can um put in some text and generate an image. Um they're not GANs, but GANs were the earliest kind of image generation models that were out there and really effective. And some of them are still used today for sure, but majority of image generation models are are what is known as diffusion, which we will study later on when we get into generative AI. But this was a huge breakthrough in using neural nets to generate images, right? That idea of generating images through neural nets, this was a big breakthrough here. 2014. Okay, and then on the reinforcement learning front, um we have the AlphaGo developed by Google and its DeepMind team. First uh kind of machine program to defeat a professional Go player. So, if you've never heard of it, Go is a board game that is thought to be more complex than chess cuz the board is a lot bigger. Um it has more complex rules to it. And more uh more states in terms of the the way that the pieces can be placed on the board, there's a lot more variability in in the pieces uh in their positions on the board. So, it's thought to be a more complex game and it was mastered by reinforcement learning using deep learning. So, this is the first time where deep learning really benefits a reinforcement learning model so much that it's able to beat Go professional Go players. Um this was a huge breakthrough. 2016, about 10 years ago. Huge breakthrough. Especially in reinforcement learning. This is where deep you know, deep learning being able to use be used in RL, big deal. It wasn't always that way. Okay, now to the big one, which is 2017. This is what I alluded to earlier with transformers. So, this is where um the very first transformer was created. We're going to study transformers later on in the course, but transformers changed everything when it came to language tasks. So, I said RNNs were used prior to transformers around 2014 to do a lot of language tasks. Transformers come along a few years later and really revolutionize everything. And all of the generative models today like ChatGPT, like Gemini, like Claude, they're all based on transformers. They are just massive extensions of the transformer idea. Um these large language models, these LLMs, are all based on the transformer idea. This was a huge breakthrough. I remember myself um I had just gotten into data science machine learning in the industry right around here, like 2016. And I remember reading this paper at the time uh and how important it was. It was just a huge breakthrough. Uh the attention it all Attention is all you need. Highly encourage you to go and read that if you can. It's it's amazing. Now, it might not make sense yet By the time we get to transformers, our lesson on transformers later on in this course, probably make more sense, but um really fascinating breakthrough. This is 2017, not that long ago, right? Really not that long ago. And not too long after this, we see the explosion in generative AI, right? Especially for text. Okay. So, those are just to name a few. Obviously, there's been other advancements over the years, but those are some of the biggest. Um and uh what we've seen is deep learning has really come a long way. There's so many cool algorithms. There's uh There's also been an explosion in in implementation uh um because we have these um frameworks readily available that are open source that anybody can use. So, more it's it's more and more people have access to these libraries and are able to develop their own algorithm in the community. So, the research community has benefited a lot, and we've seen because of that an explosion in AI tools and uh AI models because of the ease of use of these deep learning um frameworks and and all the advancements that have come over the years. So, a lot of cool uh libraries, a lot of efficient algorithms out there, and we're going to study those, all right? We're going to study TensorFlow, we're going to study PyTorch so we can build our own networks. But, the community has benefited greatly and has been part of the reason there's been an explosion recently is because of how easy it is to get access to those libraries and start building your own neural nets and trying things out. Real easy. Okay, so that brings us to uh an important question, which is why should we study deep learning? I said before that you know, it's an advanced [snorts] topic. Not everybody studies it. It's a bit of a specialization within machine learning and a lot of people just get through data science and machine learning and kind of stop there. So, what reason do we have as learners to actually study deep learning? I'm going to talk about that. One of the key reasons is that deep learning, as I said before, unlocks problems across many different domains that were that are difficult but are necessary in those domains. So, I'll give you one example is health care. There's so many image-related examples in health care that are really vital that neural nets help with significantly. So, think about analyzing images and image data of the body or of um the brain of um potential diseases. Those can be more readily analyzed with deep learning. Um and so, health care has seen a lot of advancements in analyzing that kind of data with with deep learning. Autonomous vehicles, of course, robotics, um even like fraud detection, there's a lot of advanced methodology there or even like time series forecasting has benefited from uh neural nets. So, there's so many advanced use cases that neural nets can benefit um because it can work with text, because it can work with image data. Um it just unlocks a lot of possibilities. Um so, so that's one reason is it just gives us many more problems we can potentially solve in in different domains. It's worth studying it there. Another thing is we have so much more access, right? So, um GPUs are more readily available. Um cloud environments have GPUs readily available. We're going to see that in Colab. Like we are going to be able to work with GPUs in Colab for free, but certainly like if you're paying for it on a on a cloud provider like GCP, AWS, Azure, they all have GPUs that are relatively cheap um considering that we're actually using, you know, advanced hardware there. Um so, because of that, it is because it's so readily available, we have a better chance at utilizing deep learning more than ever because GPUs are so so much uh easy to get your hands on than ever before. So, it's not to say it's perfect. Like, you know, we were joking earlier about the in in video um 5090. You know, there are some physical hardware elements that are expensive and not easy to get your hands on, but like cloud resources for GPUs um are readily available pretty easily. We're going to see that with Colab in particular. Another thing is uh you know, GPUs I've I forgot to mention this, but let me go back to the slide actually is um one of the things that Nvidia really powered was this uh CUDA framework. This is something that I mentioned a little bit earlier. This allows the Python library to interact with the GPU and be able to offload operations to it instead of having to keep everything on the CPU, which is much slower for the neural net operations that that it's doing in its layers. Um and so, in uh there was kind of this um framework to interact Python with the hardware, this CUDA um library. And um because of that, the libraries that manage neural nets like PyTorch and TensorFlow have integrated the CUDA framework and have the ability to easily manipulate operations on the GPU, um which makes working with uh neural nets so much easier. So, you can take advantage of a GPU. This is just a long way of saying you can take advantage of GPU really, really easy with the modern frameworks. So, with PyTorch, with TensorFlow they can interact with the GPU so easily because of things like the CUDA framework. Um makes it so much easier. So, the the CUDA is we don't ever see it. It's abstracted away with the framework like PyTorch or TensorFlow. And we're going to see that. We are going to build models in PyTorch and TensorFlow that will take advantage of the GPU. So, we're going to see this directly how easy it is to truly take advantage of that hardware and it's going to speed up our operations a lot. It's It's going to make it so much faster to to do that processing. Okay. So, we just said GPUs uh play a critical role um and they're more readily available than ever. So, that's another reason to kind of learn deep learning is we have access to the these resources. And it's it's really because the GPU can do parallel math operations so efficiently, way more than a CPU. So they they really We're going to see the effects of this you know, when we do especially working with um computer vision and convolutional networks, working with a GPU then becomes almost necessary. It'll speed up operations so much. Like if we didn't use it, we'd be sitting there for minutes, maybe hour waiting for the model to train. But if we use a GPU, it'll take seconds. It'll really speed it up a lot. Okay, one of the other reasons uh we've kind of hinted at this is just a low barrier to to entry. So, so this is just to say why not learn it? If you're going to learn machine learning, there's so many problems that use neural networks. And why not learn deep learning because it's never been easier to do. There's hardware available with GPUs more than ever. We talked about that. There's user really extremely user-friendly frameworks like TensorFlow and PyTorch to help us build neural networks and they're open source. So, they're easily we can pip install it, get it into our environment, and start building neural nets and training them really really easy. And we already have all the fundamentals we need to really understand model building, model training, um model evaluation. We have all that fundamentals from working with uh Python, working with uh machine learning that we've already done, data science. So, it's pretty easy to get into it at this point. So, why not? Right? It It's just going to help us uh expand our the breadth of our skills and problems that we can solve. And like I said, open-sourcing these like TensorFlow and PyTorch was a big deal. It made it so much more accessible for the community to build their own neural nets. Prior to that, it was really limited to these big companies. You had to be in one of them in order to really start building neural nets, but now anybody can because these are free and open source like TensorFlow and PyTorch. Okay. So, another key reason and you may be wondering this especially because of generative AI um taking over uh a lot of development these days, but still there's a need for trained deep learning experts and practitioners. So, many um companies still work with deep learning models and need deep learning expertise um to be able to like tweak the training, set up the training, set up the model, um make the adjustments if the training doesn't go well, really understand the model. Even if you're using generative AI, you still need to have an understanding of how the modeling works and how it functions in order to effectively use it. So, there's still a need for deep learning expertise regardless of AI and generative AI. So, it's a good opportunity. It's a It's a good skill to have. Um I can say this, like beyond just having data science and machine learning experience going through this program, I think having the deep learning experience is just another really um beneficial thing to have if you're trying to get into the industry. It It really unlocks more potential for you, I think. And again, not everybody studies it, so you're getting a kind of advancement above the competition potentially by learning deep learning. All right, let's talk about some applications. So, there's many As I said before, there's many different uh fields that really benefit from uh deep learning and working with neural nets. Um so, here's some of them, just a few, you know, um NLP being a huge one in terms of processing text, being able to understand human language. That's a big one. Uh self-driving cars using computer vision for object detection, that's something we'll talk about and study. Um not only like detection of objects, but classification of them. So, think of like Okay, here is an image and inside of it there's a pedestrian over here, there's a there's a uh a s- uh a sign, like a a stop sign, there's a tree over here, there's another car over here. Um here's the road um divider. Um so, so many objects that are that you can detect in the environment and really classify what they are, and that's all powered by deep learning. Same thing with audio processing, transforming speech to text, That's how we talked about earlier, that's how things like Alexa and those voice assistants really work. Translation of audio into text. And then uh manipulating images as well. So, generative AI with images. So much of generative AI is all deep learning based. Transformers, um diffusion models that uh generate images, GANs. All of these generate and manipulate images. They're all based on deep learning. So, so much of generative AI is deep learning based. Okay. Now, there are some limitations. Uh one of the things to realize about deep learning, we've talked about it a lot so far, is a significant amount of data is usually required. Meaning that deep learning may not be the best for tasks that have um little amounts of data. And so, what is little? Um usually we're going to be dealing with uh in the tens of thousands, if not more. That's like a That's on the very low end of maybe images or text examples in order to train a moderately sized neural net. Uh maybe even on the smaller end of a neural net. The bigger the neural net that we have, the more complex of a model we have, the more data we need. Um so, this is this is like for the problems we've dealt with so far, throwing a neural net at it would likely be overkill and would not be effective. So, it's not like we can just throw a neural net at our housing data and expect it to predict the price very effectively. It doesn't work like that. We would need a lot more data for that to be effective. Um and we would probably need to tweak the neural net uh a good amount to get an effective model there. The neural Neural nets have a tendency to overfit. So, they they need a lot of data to avoid that overfitting. They need a lot of examples. Um and and that can be sometimes difficult to come by. Um the amount of data that's required to be collected generative AI like GPT requires so much data. That's why they're using people's chats to continuously build a data set so that it can keep training it. Um so much data is required there to train a really effective model, especially one that big. Um just requires a huge data set. Um let me give an example. So, consider, you know, a model that's we've we've used this example before, right? Just differentiating or classifying cats and dogs. So, this image is a cat, this image is a dog, and etc. Um in order to to do this effectively, we would need many different images of cats and dogs. And and likely those images would have to be of different size cats and dogs, different breeds, um from different angles potentially, from different lighting and brightness of the image, um many different variations on those. And they'd also have to be labeled, right? So, it's it's still a supervised learning problem. Um so, you know, gathering that data set may be an expensive endeavor. May not be trivial. So, that is kind of a a cost burden of doing deep learning is the data. It's going to be a lot. Um you may need a really massive data set, and that can be difficult to acquire in some instances, not all, but in some cases that may be difficult to acquire. So, it's just a It is not a limitation so much as it is a caution that if you're going to use deep learning, you generally need a lot of data. And so, you have to be prepared to collect a lot and process a lot in order to um have an effective model. The other thing is hardware. We've talked about this. Now, it it's again not necessarily a huge limitation because we do have access to GPUs, but you may need to pay for it. Um and you may need advanced hardware. The more complex your model is, the more you're going to benefit from, you know, uh the more complex hardware like a GPU. Um it is more readily available than ever, but it doesn't mean that it's always going to be free and always going to be easy to uh acquire. So, there may be some work there to use that or to acquire that hardware um for a really complex model. Now, that being said, we are going to work with free GPUs for the examples we will do, but the examples we will do will be relatively moderately complex examples. They won't be massive models that would require a much more significant hardware investment to to be able to run. So, I used the example before, um something like a ChatGPT is going to be using thousands of GPUs. That's a significant investment. If you're just using one for free from like Colab to do simple examples, that's one thing, but for a really big network, um you're going to want to use a significant amount of GPU resources, um which may be costly. Right? So, it's just something to be aware of. Okay, so one of the other limitations I mentioned earlier is that deep learning, because there's um it's such a complex neural network, it can be susceptible to overfitting. Um in fact, they are very susceptible to overfitting. Can easily overfit, especially if there's not a lot of training data. And that's just the nature of the neural net. It It's so complex of a model that it can effectively memorize the training data very easily if you don't have guardrails in place. And that's one of the things we're going to study in that lesson on optimization of the training process, how to avoid overfitting. There's certainly things we can do and we're going to study to prevent a neural net from overfitting, but they are very prone to it. Naturally very prone to it because of their complexity. Right? Remember Remember we talked about overfitting? The more complex a model is, the more likely it is to overfit. The more basic a model is, the more likely it is to underfit. So, neural nets fall in that category of being really complex. So, they are very prone to overfitting. So, something we have to kind of be on the lookout for. We'll study some techniques to help us overcome overfitting or prevent it. All right. Another one that is very underrated and a lot of people don't think about when it comes to deep learning is the explainability of deep learning is very very limited. So, when we studied machine learning and we did things like logistic regression or decision tree, that was super explainable because we like in logistic regression, you get coefficients on every feature, right? You get those betas. So, we know exactly which features are important because they're going to have higher coefficients, right? And same thing with like a decision tree, you get an actual tree structure, so you know that you know the path that leads to a prediction. And so, they're really readily explainable. The issue with neural networks is they're often a black box. And And what that means is you put data into it, you get a result out, but it's almost impossible to explain every little thing that happened that led to that prediction. Very difficult to explain with neural nets because they're such a complex model. They don't have a very natural way of um having like a single coefficient. Most um to give you some context, most modern neural nets like like a uh GPT that's generating text is going to have in the billions, tens of billions, hundreds of billions, if not trillion weights uh which are, you know, think of those betas. It's going to have like a trillion of those betas. So, it's impossible to say what a single contribution to the prediction is. Now, this is an active area of research in the research community as people are trying to figure out good ways to um break down how a neural net's making a prediction, but it's still is near is pretty much not possible relative to other machine learning techniques. They're just they're really complex models. You put something in, you get something out. It's hard to say what happens in between. You know the architecture, but it's hard to say what exactly contributes to that final result. It's hard to hard to trace that back. Um so, uh deep learning can be very difficult for explainability. And And this actually has an effect in like regulated environments. Let's say you're in the finance industry and you're making um like loan decisions based on a model, um you probably don't want to use an a neural net because it's going to be hard to explain how you predicted your like uh loan or not loan decision. You You probably want to use something that is more explainable like logistic regression or a decision tree which would those you can directly see the contribution of their features, right? Leading to a prediction whereas a neural net you're not going to be able to see that. It's just too complex of a model. Okay, so that's another kind of drawback of deep learning. Now there on the opposite end of that there are cases that we don't care about the explainability at all. So you think about like a self-driving car we don't care the explanation of why this is a stop sign, why this is a pedestrian, why this object. We just want it to do that really fast. It needs to be real-time, it needs to be really fast. We don't care about an explanation in that case, right? We just need it to make an accurate accurate prediction. So there are certainly cases where we don't care about that, but in a lot of like regulated industries you may care about that. If you're if you're using a model to make some type of decision you may care about the explainability of it in which case neural nets are not ideal for that. Okay. Let's talk about some of the the frameworks then that we will use. So these are going to be basically our libraries and tools to help us develop neural networks. Basically they um they allow us to design neural networks, they allow us to train them, they allow us to debug them like how they're working or what kind of evaluation we're doing if they're producing any any bad results. And also for deployment. Now we won't really focus on deployment too much as it's going to going to be out of the scope for us a bit. We're going to focus on how to design and kind of train them. But these Python frameworks will allow us to build and train and evaluate a neural network or neural networks. And this is huge. Like like I said earlier, it wasn't always this case, right? You really had to be part of these large corporations that were doing this, like a Facebook or a Google, in order to have access to the tools to build neural networks. And now they're readily available, like PyTorch and TensorFlow. So, the main the main uh frameworks that we're going to study are uh PyTorch and TensorFlow. Now, Keras is on the screen because what we're going to learn is Keras is really um it embedded into TensorFlow. It was actually developed by Google, the same people who made TensorFlow. It's really just a part of it. It's it's an interface to core TensorFlow. So, when we hear the word Keras, we should really be thinking TensorFlow. It's the same thing. Um it's actually technically built on top of TensorFlow and allows us to interface with TensorFlow. But so so these these were built by Google. And then um PyTorch was built by Facebook, now Meta. Um but at the time, Facebook. Um And they're open source now, so they're widely available. And of of course, we will learn how to use them to build neural nets and uh uh train them. Um they're both widely used. So, both widely used in the community. Most people just have developed a preference for one or the other just based on either what they originally learned to begin with or um what they just naturally like. But the the truth is there's uh a lot of models out there, like even like an open-source GPT that exists or Llama, they're developed in both frameworks. They have a version that's developed in PyTorch, they have a version that's developed in TensorFlow. And it's they do the same things, they mostly just have syntax differences. That's the only only thing about it is they primarily just have syntax differences. But again, it's worth our time to study both because both are so widely used. Um and honestly, both are still maintained even though they're open source, both of these companies still maintain uh groups of developers that are maintaining these libraries. So there's still people within Google who actively manage TensorFlow and there's still a group within within Meta now that's actively managing PyTorch. And they they do work on it all the time, but it's also open source, so contributions can come from anywhere, really. But there's dedicated kind of people within each of these uh companies that are still maintaining these frameworks cuz they're so critical. Right? They're so critical to doing deep learning um and doing neural networks. So we're going to study them uh pretty deeply. So Keras, let me talk about Keras. It is um going to be a a interface in Python to the TensorFlow library. So TensorFlow itself, so TensorFlow was created first and then Keras came along a couple years later um to interact with TensorFlow. TensorFlow is kind of the underlying library that Keras is built on top of and it just it's much much much more user-friendly to manipulate and manage neural networks. So most people gravitate towards using the Keras library within TensorFlow um to building if they're going to use TensorFlow, they're going to be using Keras as as the interface to TensorFlow. So, it's just built on top of that by the same Google group who made TensorFlow. And it's it's the advantage of it rather than using the original TensorFlow is so much easier to get started with and build neural nets and so much easier to manage them. It's a really nice interface to TensorFlow. So, when we build models in TensorFlow, we're really going to be using Keras. Okay, as that kind of library on top of uh TensorFlow um to to build to build neural nets. So, we will study that. Is there an advantage of using Keras directly? Yeah, that's what it is. It's it's a really user-friendly interface. So, it's designed to make building TensorFlow models so much easier. Cuz the truth of it is like core the original TensorFlow is kind of ugly. It's kind of ugly and like messy to deal with. So, people when they're building TensorFlow models gravitate towards using Keras as the interface to TensorFlow because like core TensorFlow is really messy. Um TensorFlow is as we'll learn later is is basically a graph library for building like a tensor graph to to mimic neural network a graph of computation. But Keras makes it so much easier to to work with TensorFlow. So, most people gravitate towards Keras. Then just usually you won't ever use the original TensorFlow unless you're maybe doing specific research that would require it. You're mostly always going to use Keras to build your models and train them. So, that's what we'll do. Okay, so underlying Keras is TensorFlow which came first from Google. It's you know, obviously within Python, it's open source now. And this was the original library they built in-house to to manipulate and manage neural networks. You know, Keras came along as the interface to TensorFlow to make working with TensorFlow so much easier. Um, so that's something that we'll see. And actually TensorFlow has some really cool tools, uh, uh, as part of it that we will, uh, work with later on. They have tools for like visualizing the training process, which is really nice. Um, so we'll work with those. Okay. So, on the other end of it is PyTorch that, uh, that Facebook developed, now Meta, um, who still maintains it, but it, you know, has been open sourced. Um, same kind of thing. They were trying to work with neural networks and needed an in-house, you know, in-house library to do that. And so, they developed PyTorch at the time. You know, and since it has been open sourced to the community. And it's it's very similar to TensorFlow, honestly. It has the same kind of features it as TensorFlow. Um, it's just a different syntax. And, uh, so that we're going to work with both, as I said. We're going to work with PyTorch and with TensorFlow. Um, in order to build models and and, uh, train them. So, as I said, in the community, it's kind of a 50/50 split on which one people have a preference for. I personally, um, like PyTorch. I I I actually learned TensorFlow first, um, but found myself liking PyTorch a little bit more. But it's preference. Honestly, it's preference. And I've seen, um, it really just depends on which what you like better. And then, oftentimes like where you're working also determines, you know, maybe everybody has a standard for okay, we're all going to use TensorFlow or oh, we're all going to use PyTorch. Um you see things like that. But honestly, if you know one of them, you will will be well-versed in the other and and we are going to work with both. So, we'll get exposure to both. Just syntax differences, really, between the two. Okay. So, what I wanted to do was wrap up this introduction. So, we covered a lot in the introduction, but I wanted to wrap up specifically with um the life cycle of a deep learning project, which I think we're going to see mostly overlaps with the same steps we were doing for machine learning. It's just there's going to be more, you know, there's going to be more details that are that are involved here when we do training, when we do uh model building. But mostly, the life cycle this is honestly, this is the same as machine learning from a high level, right? It's mostly the same. We have some type of data collection and data prep phase, which is kind of what this is. So, so this is like our data prep that we do for a machine learning. We're used to doing that um first, usually. Now, prior to that, there may be some planning that has to be done uh to collect data. And in deep learning, that may actually be a significant amount of work to um go out and gather a bunch of images or process a bunch of images or text data, you know, and label it appropriately. That may actually be non-trivial, but uh you know, all the examples we will do will assume we already have that data. But other than that, it's mostly the same steps in term at a high level of like once we have our data prepped, we go ahead and and um train we build and train our model. Um obviously, the details of doing this with a neural net are going to be much different than with a basic model like logistic regression or something. Um, but once we train the model, we then evaluate, right? We evaluate it on some test data, evaluate, and see how it performs. And that And the reason there's an arrow backwards is based on that evaluation, we may need to make make some tweaks. So, you know, at each one of these, it could be very iterative. Um, in the training, like if we're not if we're underfitting, we may need to go back and revise our data. We may need to revise our model even. Um, so it can certainly be iterative. And then, after you have a model you're very satisfied with in the evaluation phase, that's where you can get into kind of um, deployment of it and then like monitoring it and main maintaining it. These two things were not going to be focused on too much because, again, it's going to be a little bit out of scope for us. Um, we're mostly going to focus our attention on these two phases, training and evaluating of neural nets. Okay. But, let's talk a little bit more about these steps. So, the planning, uh, as I said, this is where, um, the primarily, whenever you're doing deep learning, this third bullet is is really critical. So, I would lean on this bullet as really critical as So, planning really meaning what resources we're going to need in terms of data. What kind of data are we going to need? Like images, text, you know, what are we working with here? Not only data, but hardware. Like can we get access to like are we going to need a GPU? Um, can we get access to that? For especially for deep learning, that's going to be really critical to know your resources ahead of time. Um, and be able to have those available, especially the hardware, um, like a GPU. So, usually there's some thought put into that from a planning perspective, um, in terms of the resources. Uh, then there's the actual collection. So, depending on the problem, this could be very different, right? We may be physically taking a bunch of images, we may be downloading a bunch of images, or or grabbing them from some database. Um, we may have IoT sensors that we're collecting sensor data from, or like our self-driving cars taking a bunch of imagery, sensor data. Um, so, that could be different, um, audio, you know, audio or video. We could be collecting, for example, we need to collect that, and we also need to label it, right? We need to have adequate labels. And, like, think of the cat and dog images, right? We need to take every single image and provide a label with it, um, if it's a cat or it's a dog. We also need to gather those images, right, in the first place. So, there's going to be data prep that goes on here, no different than before. Okay, then we're going to talk about the training phase. So, one of the things about the training phase is that's not really shown here, but is definitely a prerequisite, is we first need to build Obviously, we need to build, I would say build / architect architect uh, the model, right? The neural net. Neural net. So, obviously, we need to do that, and that will feed into the rest of the training process. Now, I, um, will talk about this later. I'm not too concerned about these steps. These are just the steps of the training that we're going to learn. Um, so, this this will really go backward and iterate through these steps. But training of a neural net is very unique. It basically involves sending a bunch of data through the network, figuring out what your error is, and then going back and updating the network. That's essentially what all these steps do, and then you iterate through that on your data over and over and over again until your network uh learns how to effectively make predictions. Um, so it's a very iterative process, which is what makes it take a long time. You're going to iterate through all of your data, and you're going to do that on a number of steps, which could be thousands of times that you're doing that. It and it's really the same thing over and over again of again, sending data through, computing how far off you are in the error, using that amount of error to go back and update all your your model. Um, this process is where that gradient descent is going to come into play. So we kind of touched on this in the recommendations with that matrix factorization learning. Um, we are going to study this in detail quite a quite a bit more here in neural nets because that's really what's going to power neural net training is um gradient descent. So again, these things I don't expect us to know right now. Just giving you a preview of the training is a lot more involved. There's There's a significant amount of computation involved here in like sending data through, generating predictions, computing error, updating all of our weights in our network, which could be billions and billions and billions of weights. So this this backpropagation step could be significantly computationally intense um in the gradient descent. And you're doing this many many many times in iteration uh across all your data. So, it is significant the amount of resources that are spent in training a neural net. Okay? All right. So, there's That's the training. And again, we're going to have a lot more to say about these in future lessons. So, let's put these terms on hold until we learn more about neural nets. All We'll cover what forward propagation means, what loss means, what back propagation means. We're going to study all of those. So, a lot more to come on those, but for now, just think of the training phase as we have to iterate through having our network learn and adjust, learn and adjust, learn and adjust over and over and over again until it learns effectively. Okay. Um so, as I said, the model is goes through iterative uh performance um evaluation essentially in terms of how much error it's generating. And so, we keep an eye on that and continue to train it over the course of the training phase, which may be lots of iterations. And you can go backwards at this point. So, based on the training, you may go back and adjust your model, you may go back and adjust your uh parameter like your hyper parameters of the training, which is going to be a few. You may go back and adjust your data, which is what this slide saying is that we may have to go back and collect more data, we may have to double-check our labels. Um there could be a lot of reasons that we need to go back and make adjustments. It could be we change the architecture as well, like we actually change the model itself, the neural net. Um so, going back to this diagram, there's a lot of iteration that happens. So, you know, based on the training, we may go back and alter our data, we may alter our model based on the evaluation, we may go back and um alter our training hyper parameters. A lot that we, you know, may be modifying during the course of the training. >> [clears throat] >> So, the training is going to be more intense, I think, in neural nets than what we've seen to this point. With Usually, what we've seen to this point is we just run a dot fit, and everything happens for us, right? Especially with scikit-learn, right? We just run dot fit, and boom, we have a model that's trained. Like a logistic regression or uh decision tree. Right? Dot fit, and it's done. This is not going to be as simple. It's going to be a little There's going to be more setup, more moving parts involved, um more computationally intense, for sure, to do to do the training. Okay, but either way we slice it, there's still going to be an evaluation phase, meaning that, you know, even if we train, we still need to compute metrics to see if our model has performed well enough on the training data and on this testing data. So, from a supervised learning perspective, that step will basically be the same as what we've seen before. Once we've trained the model, we're going to apply it to a holdout test set of data, and decide whether or not the model's performing well. Is it overfitting? Is it underfitting? We can tell that on a test data, just like we did with machine learning. That will not change. So, what we're going to do is still have training data >> time. It's a neural net. >> Okay. Then, if everything checks out there, we can deploy it and there's many different ways to do that. Not going to get into them in this course, but uh suffice to say that once we have done the evaluation, it's kind of ready to go. And we would do a lot of um kind of packaging up and infrastructure and things to kind of host the model. Um many different ways to do that, but the goal here is to be able to actually use it to make predictions in in some some way. Right? So, maybe we're using it in like an object detection inside of a self-driving car or something. And of course, once you have that model out there, you're going to keep monitoring it and maybe make adjustments as you see the predictions coming in. So, one of the goals that of even building the model in the first place is of course, we're going to apply that to new data that has never been seen before. And that's where things can go off the rails. So, you really got to monitor the performance as you are predicting on new data, right? And see how it's performing. So, um you may want to retrain it, make adjustments, maybe in the extreme, maybe you have to go back and uh change the model up. Um but you should be keeping an eye on the performance if you are using that model on new data. Okay. So, uh just to wrap up here on this introduction lesson. So, that that's everything we've covered so far is just background information about neural nets. So, we have not talked about what a neural net is technically from like a math perspective. We haven't covered um you know, how we train it. Um that's all to come. This is just background, right? So, we've talked about deep learning being a specialized subset of machine learning that particularly focuses on using neural networks. We've talked about that. And we've also talked about it's really well suited for unstructured data like images, text, audio, video. Um so it's really really good at that. However, it requires usually a lot of resources to function. So it requires a lot of data, a lot of computational power like a GPU to be effective. And then in terms of the life cycle, mostly follows the same steps as machine learning. It's just those steps are going to be different because we're dealing with a different kind of model, right? A neural net. Okay. All right. So we're going to start lesson three. So if you're following along, you want to go to those uh notes and [snorts] go to lesson three. So we're going to start our study of neural nets, and particularly with the most basic kind, starting with a perceptron, really like a simple model of a neuron. And the idea is to build up from there into uh bigger and bigger neural nets. So what we're hoping to learn here is basically get into more of the details of how a neural net functions. So structure, functionality, um including starting with the very basic perceptron, and then expanding that a little bit into multi-layer perceptrons, and then talk about some of the um advancements beyond there that are out there like a deep neural net that that extends that. Things like CNNs, RNNs we'll mention as well. And then talk about some of the activation functions that are uh present in neural nets. Now it it won't make sense this moment when I say activation functions, but they'll make more sense when we get into uh what's involved in a typical neural neural net, especially modeling a neuron. Um there's things like activation functions that are really critical. We'll study some of the most popular ones, um like ReLU, sigmoid, softmax. Um and then we'll talk about some of the uh some of the issues that can pop up during training in particular. Um, things like vanishing exploding gradients. Okay. So, lots to cover. Um, let's start with modeling the neuron, right? So, let's start there. Um, so obviously the models that we're going to build are networks networks of neurons and we have to start with kind of the most basic model of a neuron and it's very much inspired by the biological neuron that is uh within the brain. And uh so, not that we need to know all of these things about it from a biology perspective, but just to know that the model of the neuron is obviously based on how a biological neuron kind of functions. Um, so from this perspective, you know, there's inputs that are sent through, they go through the various um nerve cells and then out comes some type of output signal that flows through to the rest of the network. That's going to be very much how the neural networks will function is there'll be kind of input data flow through this collection of neurons and then out on the other end comes this output data. So, we'll see that as we go. So, in a biological neuron there's these various components like the cell, a synapse being a connection between two cells, axons transmitting the output, dendrites receiving input. We're going to have equivalent components in the artificial neuron model that we are going to take a look at such as the perceptron. So, in the artificial neuron, we have a similar setup. Now, this is really really critical to understand is that the artificial neuron is very much similar to a biological neuron in which case we take all of the inputs and combine them together to generate some type of output. Now, the basic model of a neuron is going to do this really important process of taking the inputs to the neuron. So, those could be, let's call them X1, X2, and X3 or more generically, however many there are, Xn. And then waiting those with different weights. Okay, waiting those with different weights, and then basically taking a weighted sum. So, we have something like W1 uh X1 + W2 X2 + blah blah blah + Wn Xn. Now, what does this remind us of? Hopefully, this reminds us of something like a linear regression, right? It's very similar to that. That that kind of does um remind us of a linear regression. The difference is that the neuron is actually going to process this weighted sum through what's called an activation function. So, every neuron is going to be activated or deactivated based on the this flowing through some activation function. So, you take that linear sum, you pass it through an activation, and often this activation, so this F F is called an activation function. Which gets its name from the biological neuron of, you know, a neuron activating or not. Like or some people call that firing, right? A neuron fires. Um it activates. So, this this F is called an activation function, and essentially what the activation function is doing mathematically is applying some non-linear function. So, it's typically the activation is going to be non-linear in nature. It's going to be some type of function that is non-linear. Okay? And it's going to um it's going to be applied to that weighted sum of inputs to that neuron and then produce some sort of output, which is the activation value. Um it's you know, producing some type of output. So, in this case, the neuron model, this is the this is the model of the neuron, is really just applying a activation to some weighted sum of inputs flowing into this neuron. Um so, in this way, it's kind of like the biological neuron where we had input signals flowing in, input signals flowing in, and then some type of output signal coming out, right? It It's basically that. And then what the neuron is doing is doing an a weighted sum through an activation. That's all that the neuron is doing is doing the weighted sum, summation of weights times inputs. So, you're have you have your signals X, you generate an output, which is um the weighted sum through the activation, and you get this output signal uh Y here. Training the model with lots of data determines Yes, exactly. So, during the course of um doing the computations, the neuron will will be active or not active. Um it it's really not a matter of binary like active or not active. It's more of what is this value that comes out of this activation function? That's the output. Right? What is that value that comes out of the activation? This is this This is this value here. It's that weighted sum of inputs through an activation is really the model of the neuron. Right? So, what is that output is really what we care about. The reason we care about that is because I want you to think about where this is headed in a in a network. So, in a network of neurons, what we're going to have is a collection of these guys. So, we're going to have many more neurons that would fill up a layer of neurons. So, this is just one. So, we'll have a lot of these guys, and each one of these neurons could produce of outputs. Right? They could produce a collection of outputs um from their layer. And so, and then what you what happens is these outputs become the input to the next layer. They The signals keep flowing. Right? They go into the next set of neurons. Right? And then those um What actually happens is you start to build this network. So, not only does this go like this, but there's many more connections. So, like this will become an input into this. This will be an input into this. This will be an input into this. And this will be an input into this. And same thing over here. Like this will be an input into this. This will be an input into this. This will be an input into this. And this will be an input into this. So, everything from the previous layer which is coming out of that neuro all those neurons will flow forward into the next layer of neurons. And on and on and on, however many layers you might have. So, you can have a significant amount of these layers of neurons in a really deep network. You might have lots and lots and lots of layers. So, we're going to study that. We're going to stay study that as we go along, but that's really what we're building up to. Yeah, that's that's what we're building up to in a neural network. That's why it's called a network. It's really laid out in the structure of neurons sending data forward into the next layer of neurons and on and on and on until you eventually reach a final output. Now, I want to give you a basic use case of this neuron. I want to give you a basic use case of this output because it may be it may be a little vague on what that output is. Most of the time for a neural network, this output will generally be a probability. So, this will generally be some type of probability, especially for a classification. Probability will usually be a probability. So, especially for classification. So, let's say let's say we were predicting if it if an image like with with a brain scan is cancer or not cancer. This would be like a probability of this would be like a probability of cancer that we generate like in a binary case, right? Or spam or fraud, right? Generally represents some type of probability. It doesn't have to. Like if we're predicting a regression type of problem. Like let's say we were predicting the price for our housing price data. This this could be a price, right? This could be like a full value some some floating point number price, right? That that that could be possible. But, it's a single output from this neuron that could represent typically for classifications be a probability, but could be a price like in a regression problem. Okay? So, again, just to recap this cuz I think this is really, really critical to understand moving forward in neural nets is what does a neuron do in the network? Every neuron is processing a weighted sum a weighted sum of inputs W2X2 WNXN. It's a weighted sum of inputs through an activation function F. That's what every neuron is doing and that produces this output value from the neuron, which is that activated value, right? Some non-linear function. And by the way, when we train the neural net, what we're actually learning during the training process, what gets updated during the training against real data are these weights. These weights get learned similar to when we do linear regression, right? We learn all those betas. Same thing here, we learn all of these weights. Now, do you guys see how if we have a really big network, there's going to be a lot of weights? So, imagine we had a really big network with lots of neurons and they're sending data like they could take a take an image, which is an image could be like 1,000 by 1,000 resolution. Um that's so 1,000 by 1,000 is going to be our input. And then, essentially, what we're going to do is take those, pass them through this network, and generate some type of classification of like cat or dog, some probability of like might be cat, it might be dog, it might be giraffe. Um so, maybe the output's actually three. So, we have three final outputs like cat probability, dog probability, or giraffe probability, but there can be a lot of weights because we're going to end up with this really massive network where everything is connected together, right? So, there can be a lot of neurons, um a lot of weights that are that are connecting layers to each other. So, these networks can grow massively with the amount of weights that have to be learned. Um now, in this simple model of a neuron, there's only three weights. Only three weights that we would have to learn, so pretty simple. But, in like a chat GPT, there's 100 billion weights. That would be pretty conservative, actually. Um their open-source model, their transformer network open-source GPT is around 20 billion weights. Okay? So, just to give you an idea, there's 20 billion weights that it would have to be learned. So, you can see why substantial resources would be taken to train a model like that. There's going to be 20 billion of these things that have to be learned. Okay? So, let me pause here. Any questions about this model of a neuron? So, so mathematically, a model of a neuron is just taking a weighted sum of previous layer inputs, passing that through an activation to generate an output from a neuron. That's all it's doing. Okay, that and that's that idea is is called the perceptron. That's basically the perceptron is taking a weighted sum, passing it through an activation. That's essentially the perceptron. We're going to study that next. But, this is the most basic form of neural net computation you can possibly have, just a single neuron that's taking a weighted sum of inputs and passing it through an activation. So, these inputs could be features, they could be pixels in an image, right? Many different things. Um but you're just taking the weighted sum of those, passing it through activate passing it through some activation, um and generating an output. One of the questions that we should have at this point is what is the activation? It it The activation is actually something that we can change. We can set what we want it to be depending on the problem. So, some activations are going to be going to be pretty popular in some cases. In other cases, we'll use other activations. We'll learn which activations We'll actually talk about it later in the lesson. Um what this function usually looks like. Um in terms of taking that weighted sum, passing it through some type of function. That function might change from neuron to neuron in the network. But right now, we're just keeping it generic as some function. Some non-linear function that this weighted sum goes through. So, in that case, you know, it's different than a regression. A regression would just take this and be done with it, right? It would just take this weighted sum and that's it. That's the that's the answer. But for a neuron, there's one extra component to that, which is taking that and passing it through a function. Right? So, so a little bit more involved there. Okay. So, uh everything I just said um is kind of depicted here where we are taking the um the the weighted sum. Now, there may also be a bias. I forgot to mention that. Um similar to what we had in linear regression, right? There could also be a bias. So, what we're really doing here is taking some function. So, this this may represent the function that we're doing, some step function, um and generating an output, which kind of mimics the biological idea of an activation of the neuron, right? So, if the signal is big enough, if it reaches some mathematical threshold, um then the output signal is generated, meaning that neuron is activated, and there's some some output that's generated. It's going to be the same here. We're going to be taking a weighted sum, and then passing that through some activation F. So, basically what we could do is think of this as some intermediate value Z is the weighted weighted sum, and then we're taking the activation of that Z and generating uh an output value Y, right? Which which the activation is going to be something like this. Uh maybe a non-linear step function, maybe an exponential. Um it just depends. It depends on the situation. We're going to use different activations. So, that is the artificial neuron model, um weighted sum through an activation. That's [clears throat] actually it for the neuron. So, some terminology in terms of the difference between the artificial model, weighted sum through an activation, um and the actual biological model, or the biological neuron, um the nucleus is known as kind of the node, which is the node is just this um the node really is just like a graphical kind of node here that uh we could imagine as part of a network. So, a network's going to have lots of nodes uh to it. Um we have our inputs, which are coming from usually a previous layer in the network, um but, you know, in the in the case of a simple neuron like this, think of it like our features, right? Our features, like our our bedrooms, bathrooms, square footage from our housing data, things like that. Um we have weights, which are kind of like our synapses in the biological sense, and then we have axon, which is the output. So, there are kind of mirrors to the biological definitions. Okay. All right, so let's expand on this idea from a single neuron into a network of neurons. So, a neural network then is just the extension of this idea where now we have interconnected layers of neurons, which is basically our computation model for the overall brain, basically a whole network of biological neurons working together. Um each network consists of uh basically an input layer and an output layer, and then a bunch of potential layers in between. So, you can have as many hidden layers these layers between the input and output are known as hidden layers, and there might be many of them, potentially many of them. It depends on how deep the network is. A really deep network depth is really based on how many hidden layers you have. So, if you have very few of them, it's going to be a shallow network. If you have a lot of them, that's going to be a deep network. So, this each one of these blue circles is a neuron. So, this is a neuron. And you have many neurons connected together, but every neuron's doing the same thing. It's taking this weighted sum of of inputs and passing it through some activation and generating an output, right, that um goes into the next layer. Does the activation of a neuron depend need the inputs to be above a certain threshold? Um biologically, yes, but in the in the neural network model, no. Um we just activation really just means taking the function and applying it to the weighted sum. That's all we mean. Uh so it's not it's not required to be above a certain threshold to to be activated. We we don't really care from a biological perspective like activated or not. We care more about what is the output, which is this quantity. We care about this output, right? Uh the the weighted sum through the activation. It's just it's a function. Yeah, the activation is really just a non-linear function. We care about the output, exactly. We care about the output that's coming out of that neuron. Because what comes out of Does it make sense that what comes out of that neuron is really important for the future layers, right? Cuz like what comes out of that neuron is going to influence in the network, right? What we produce Because what we produce here flows into the next layer and on and on and on and on, right? So whatever we produce from one layer becomes the input to the next layer in the network. So it really matters what comes out of a neuron because that's going to be an input to the next layer. Really really matters what what the output is of every neuron. Yeah. Okay, so just to go back to this definition. Again, the network is this whole thing. So it's all of these neurons connected together and a a collection of neurons that are kind of forming an input prior to another layer. It This is basically one layer, right? This is a layer of neurons and they're and when we build our networks, we can actually control how big that layer is. So how many neurons are in there? We can actually specify that when we architect our network. Um that's something we'll be able to do. And how many outputs do we want? So, if we're the outputs are usually going to be relative to how many classes are we predicting or how many outputs do we generally want? So, in like let's say we're predicting between is this image a cat or a dog, um we we would only have these two outputs. So, this would be probability of dog and this would be probability of cat, let's say. Or if it was like if it was um if we had a giraffe, then that would be a third member of the output layer. So, I would need to generate a third for a giraffe. Right? Or in the case of ImageNet, they needed like in the ImageNet, remember they're predicting a thousand different categories of items. So, they need an output layer that has a thousand neurons in it. So, they need a thousand neurons in the output because they're predicting a thousand potential categories of images, right? So, that output layer should match what your final prediction is going to be. So, if you're predicting 10 categories, you need an output layer that has 10 neurons, which are going to represent the probabilities of each class. If you're predicting a regression, you only need one output, like the price or the temperature or whatever it is, right? You only need that one output. Or if it's binary, like fraud not fraud, cancer not cancer, you may only have one output, which is the probability of uh of of the like cancer or fraud or whatever it is. Okay. So, how does a neural network process data and make a prediction? So, it's kind of what I've been saying. Inputs go through the very beginning layer and then um every neuron that's in that layer receives inputs, assigns a weight, passes that through an activation to generate an output. So, every neuron generates an output. And the outputs from that first layer are then forwarded to the second layer for processing in the same exact way. So, you take a weighted sum of all those inputs from that layer, generate an output, pass that along to the next layer, and on and on and on. So, this network, you know, could have many, many, many layers. So, there could be another layer here that's another, what we would call, hidden layer, which is anything between the input and output. And what we should realize is there's going to be contributions from every neuron that flow into each one of these neurons. Right? So, there's a lot of connections there. Every connection is weighted. Right? There's a weight. Every Every connection contributes some weight to that neuron, and we process, as usual, we process a weighted sum through an activation of all of these guys. It's just that their inputs are going to come from this previous layer. Hopefully, that makes sense. The inputs are going to come from the outputs of those neurons from the previous layer, and on and on and on. So, in this way, data, we would say, is kind of flowing forward, or what we would call forward propagation. We forward propagate because it's it's literally like a propagation, like a signal propagation. So, forward prop, or forward propagation, because data is flowing from input all the way to the output, going through all these neurons. So, that that's how we make a prediction is we forward propagate inputs through the network and generate an output. Right? So, that's known as forward propagation is sending data through, generating a prediction. Um which could be a prediction could be probabilities, right? There's 80% probability this image is a cat, 20% probability this image is a dog. So, therefore, our final prediction is um, cat. For instance, so this could be 0.8. This could be 0.2. And this would be 0.8 probability it's a cat and 0.2 probability it's a dog. Right? Something like that. And we we're generating that by taking our image and flowing it through forward propagating it through this network, generating these probabilities. That's just an example. Okay. Okay. So, where we're going to start uh, with in terms of these networks, the very first one that we've actually already been studying is the perceptron. So, the perceptron is what is exactly what we've been studying. It's just the model of a neuron. Of a single neuron. Um, we will graduate from that into multi-layers of neurons. So, um, multi-layer perceptron. Um, so more than one layer, more than one neuron for sure, but potentially more than one layer. And then we'll graduate from that into deeper networks that have many layers to them. Uh, and then and then once we do that, we'll be in a good position, especially once we learn how to train these guys, um, we'll spend a lot of effort on training these and optim- optimizing that training process and building it within our code, like PyTorch or TensorFlow. We will then start to apply variations on these architectures that involve convolutions. So, the neurons will go from simple weighted sum through an activation to a completely different type of computation that the neurons are doing, which would be like a convolution. So, in in the um, just to go back a step to put this in perspective, in this picture, we're doing what we would call perceptron-style calculations right now. Meaning, we're just taking a weighted sum, passing it through an activation, and sending that forward through the network. When we get into computer vision and working with text and RNNs, these neurons will now be way more advanced. The neurons will do things like convolve or do convolutions, which will be um a very sophisticated computational operation that will require a GPU, or else will take forever to train any models. So, we will graduate from these very basic neurons. So, right now, we're just talking about a basic neuron that's doing this, weighted sum through an activation, and will graduate into convolutions, and then will graduate into RNNs. So, doing recurrent calculations, uh which mimic kind of sequence memory um for for learning how to work with text sequences, like sentences. So, a lot to cover, obviously, before we get there, but right now we're starting with the most basic model of a neuron, which is this perceptron kind of style calculation, uh just sending a weighted sum through an activation. That's it. But, eventually, these neurons will become more and more complex, like doing convolutions for image data, or doing RNNs for uh text data. And eventually transformers. Um so, so these neurons will do even more advanced things, like attention in a transformer. Okay, so a lot of exciting stuff. Talk about the uh you know, the perceptron um being the simplest type of network, that's where we're going to start. It's mainly used for binary prediction. So, because like we can just produce that one output, which could represent a probability. So, the perceptron was the simple model of the neuron and it was originally used for binary classification problems. So, that's where we'll start. It's the same thing kind of for binary. Um and the perceptron's really only good if your data is linearly separable. We'll talk about what that means and why that's really only the case. Otherwise, we need to go to more advanced like more layers. So, it the perceptron's going to be kind of limited in the sense that it's a single neuron. It's not a network. So, it's not going to model too many problems that well. Um but it is the foundation for building on that and generating many many layers of neurons that that we will see for larger networks. So, and that's where we go with the MLP, which is multi-layer perceptron. Um basically multiple layers of these uh perceptrons, of these neuron models, and uh it can handle a little bit more complexity by learning non-linear things. Um because it has multiple layers passing those inputs through these non-linear activations, it can learn more effective. However, the MLP is not necessarily a deep network. So, it's not going to have that many layers of perceptrons. Maybe only a handful. It's not going to be that deep of a network. Um so, itself will be kind of limited and especially won't work well for like images or text. But for maybe certain problems like our housing data, might work okay. But we'll certainly graduate from perceptron to multi-layer perceptron as a little bit bigger of a network. Okay. And then we'll work our way up to deep neural networks, which are going to have many hidden layers. So this we we've had this image before. Here it is again where, you know, a deep network's going to have This is the depth here. How many hidden layers do you have? A deep network's going to have many many many layers. Okay. So again, like modern generative AI neural networks are going to have thousands of layers. Thousands of these hidden layers here. And And notice how they're all connected because, you know, the inputs Sorry, the outputs of one layer become the inputs to the next layer. So you can see connections from each of these um contributing to that neuron. This neuron is taking a weighted sum of those and then generating an output. And then there's connections from that output to every other uh neuron. And on and on and on, right? So this big network, you get um a certain number of outputs a certain number of outputs that um mimic what you're trying to predict. So again, there's like this could be a thousand for the ImageNet problem. Um it it let's say we're doing cat, dog, or giraffe. This would only be three. Sending an image through, forward propagating it, generating these probabilities, which is our prediction. That's kind of how it works. Is it always left or right, or can it bounce back? So it is left to right for the networks that we're talking about. The only um time you get the circle like is a recurrent neural net, an RNN. That is the So the circle you're describing is the essence of a recurrent connection. So it has the ability to do to do this, which is kind of like mimicking a memory of a sequence sort of. Like going back and and remembering what our previous inputs were. Otherwise, it's going to flow forward. It's going to be forward propagation left to right. But in an RNN, you would get connections like this, like you're thinking about. I think that's what you're talking about. Those recurrent connections. We would get those in an RNN. Yeah. So, not Yeah, we won't have those yet, not until we get to RNNs later in the in the lessons. So, and then the the the So, we're going to study this. We're going to work our way up to this, and then as I said, once we learn how to build these guys, these deep neural networks, and really work with them in PyTorch and TensorFlow, we'll graduate to CNNs, the convolutional neural nets, which was this was motivated by mimicking kind of like the visual reception of the of the eyes in the cortex. Using the convolutions to to accomplish that. Convolution being a a type of computation will basically replace those neurons with more advanced computation. And the visual reception is kind of mimicked by the convolution, allowing us inside of images to pick out things like objects, things like shapes, edges, lines, human features like faces. Allowing us to learn those features of an image. So, the convolutional net, again, this was that AlexNet 2012, really revolutionized being able to learn uh how to predict on image data. So, we will learn about CNNs after we cover deep neural nets. Uh IT stands for I'm not sure what it means in this case, actually. Not 100% sure. Yeah, not not 100% sure. I'd have to look that up. I think they're just different um sections of the brain. I think they're they're short for like different uh sections that process like uh visuals. Okay. There we go. Yeah, they were different regions. Inferior temporal cortex. Nice. Okay. That makes sense to me. Okay. Okay, so then talking about RNNs really quick. Um so RNNs are a lot you know, networks that we're going to study later on in one of our lessons after we cover deep networks that will handle sequential data, right? So this is going to be things like language because as I said, that's a sequence of words one after the other in a in a sequence. Um so we will handle text with RNNs. Also, um a a very popular use case for RNNs, especially these days is like forecasting. So time series data like uh sales or revenue day by day by day, you can you can build forecast with neural RNNs um because they're really effective at sequence learning. Mainly because they um allow for these recurrent connections. Uh so they kind of have this short-term memory, which is effective for some language tasks. And for forecasting, unfortunately, that short-term memory is is too short. It's not able to learn the type of context that we see in like a a chat GPT for instance, with a very long prompt or a lot of text, which is what we see in transformers. So RNNs is kind of limited relative to transformers on language tasks. Mainly because that short-term memory is just not nearly what it needs to be to generate text or do effective translations, things like that. So, we're going to learn about that when we cover RNNs. Um one of the things that I mentioned that's certainly true is when we uh work with neural nets, we will be able to customize basically everything about that network. How many layers there are, how many neurons are within each layer, what those activation functions are. So, you can think of like you know, this is going to be completely customizable from it's same with input and output. How many uh inputs are are there? How many outputs should there be? Like if I'm predicting three or 10 classes, or maybe two, or maybe just one in a binary case. Um basically everything about this is going to be customizable when we build our models. And and that's just what we would call the architecture of the network. Right? How many layers there are, how many neurons there are within each layer. And you can see that like when you have a deeper network with a lot of different uh layers here. So, like this example on the right, you can see this is going to have a lot more um this has a lot more weights to it that have to be learned. So, it's going to take more data. It's going to take longer to train a model like this cuz there's so many more connections, right? There's so many more weights um that have to be learned during this process um than than if you had fewer layers. So, um that's a trade-off, right? Maybe you need more and more layers to learn a really complex problem. That may be necessary, but you're going to pay the price when it comes to computation cuz there's just going to be more weights that you have to keep track of, right? And and and learn. Um so, this is something we'll see when we get into TensorFlow and PyTorch, we're going to see how do we build this? What's the code that generates this kind of network? That's something we need to learn. Right? What's the code that does this? That's something we're going to learn. It's It's actually going to be relatively easy inside of TensorFlow and PyTorch. They make it easy because those libraries are so good at building it. So, not only that, like we'll control the architecture, but we'll also control the training and the evaluation. So, all of that we'll be able to set up um inside of our our you know, PyTorch or TensorFlow when we get to that. Okay. So, one of the things that's true about neural nets is they typically require more data, and um there's uh there's kind of a graphic here to show that like if you have even a small neural network that's very shallow, so small being like number of hidden layers, you know, it can still outperform traditional machine learning, so think of like logistic regression, decision tree, random forest, XGBoost. It can still outperform all of those. It's just going to require more data. And the larger the model, the more data that you're going to need. So, you can kind of see that on this graph, right? The large neural network has a lot of potential to perform really, really well, um but it needs the most data. It's the larger that the network is, the more data it needs to to function properly. Now, these are relatively close, so just a shallow network isn't in especially in the small data um down here, it it's not until this point that um the the even the shallow neural network kind of surpasses traditional machine learning. So, again, for small data, um it may be desirable to just continue to use traditional techniques like what we've studied. But, as we get larger and larger data size, um if we're working especially with unstructured like images, text, we're going to need neural nets. And um you know, usually we'll start small and work our way up to larger neural networks if we need them it you know, in terms of the architecture. Which is how many layers do we have, how many neurons do we have, those kind of things. Okay. So, the the depth of the network, like how many layers we have, is often um relative to the task. So, if we're doing, you know, working with images like in an image classification or like object detection or like in a in a transformer for a generative AI use case, that's going to be a much bigger network than maybe a simpler modeling task like predicting the price on our housing data where it's very structured, has a certain number of features to it, we may not need as deep of a network. So, this is again where we will uh we will, you know, practice building networks and seeing when that makes a difference. Like how much we change that architecture, uh how big of a difference does that make on our modeling problems. We're going to see that as we get into it. But, here's a good you know, this is a good picture here on the uh this one has two layers and it produces two outputs. Um so, it takes four input features, produces um two outputs which could be probabilities, let's say. And you can see like these these symbols are exactly what I've been describing as you take a weighted sum and you pass it through an activation. That's what the sigma and the F is. So, the sigma is like the weighted sum of the inputs to that neuron. [clears throat] Every one of these circles is a neuron, right? So, this is a neuron. This is a neuron. This is a neuron. Neuron. Neuron. Neuron. Neuron. And um technically all these other circles are neurons as well. These are just output neurons and these are input neurons, so they're not really doing much. They're just processing the input or holding the output. But but they're going to be a weighted sum through an activation for all these neurons, right? Weighted sum through an activation. That's exactly what what this is meant to represent. Sigma weighted sum through the F, through the activation. Um [clears throat] so, let's talk about the perceptron uh as a a model that is based on that neuron that we've studied so far. Um basically one in the same. So, what is What do we mean by perceptron exactly? So, I've been using the word perceptron a little bit. Let's actually define what it is. Essentially, it is a very basic neural network that is designed for binary classification. So, think spam not spam, right? Fraud not fraud. So, it has only two choices. And essentially what it is is the neuron such that it has a very simple activation. This is the activation right here, which is that um you get a a classification of one if your weighted sum times your your inputs plus your bias is positive. So, this has an activation function like this, which is basically just checking is your weighted sum uh is your weighted sum times your inputs or weighted sum with your inputs um greater than zero. So, if it's greater than zero, we're going to output for that neuron a one. If it's negative or equal to zero, we're going to output zero. So, it's In that way, it's just taking that model of the neuron. So, you're taking all of your inputs, your X X1, X2, X3, X XN, however many you have, and you're feeding all of them into the single neuron, which is taking a weighted sum, and, uh, taking a weighted sum of those inputs, and then passing it through an activation. We're just going to produce a a zero or a one, right? Um, a Y, which is zero or one, depending on this activation, which is, um, was it positive or was it zero or negative? Right? Is essentially what this activation is. So, it's a very simple model used for binary classification that's basically just a neuron, um, just a single neuron, taking that weighted sum, uh, passing it through the activation. This is the activation. Activation function here, which is basically just checking, is it positive, you get a one output. Uh, is it negative or zero, you get a zero output, right? For the for the neuron. That's all it is. So, a perceptron a very simple model, um, that is based on this neuron calculation. The thing that's unique about the perceptron is it's just using this particular activation, um, to see, you know, do we get a one or a zero for a binary classification? So, what's interesting about the perceptron is, um, you know, it's, uh, using the basic idea of a neuron in every sense. It's taking a weighted sum through an activation, and what gets learned in this perceptron is the same thing that we've talked about learning in any neural network, which are these weights. Right? These weights are going to be what gets updated through the training of the perceptrons. We need to talk about how does that happen? What kind of adjustments do you make? Um but essentially, we're the algorithm is going to learn those weights of that neuron. Um so that it can produce good uh outputs for your data. Right? You want high-quality predictions that line up to your your like spam, not spam, fraud, not fraud kind of uh uh predictions. So it's exactly like this picture. Uh it's just the output again is going to be a binary zero or a one, and that activation is going to be particularly checking if it's positive or if it's negative or zero. So it's exactly this picture we just talked about from earlier with the neuron. Um for that reason, you know, the weighted sum through an activation is often just referred to as a perceptron-style computation, cuz that's essentially what it is. The perceptron's just using that particular activation of seeing if that weighted sum is positive. So it's that very particular activation. Again, I'll draw it here. You know, we we do this uh weighted sum, and we're passing it through the the activation, and the activation is such that um if [snorts] this weighted sum is positive, then you get um then you get a a one. Um if it's uh else, basically, you get a zero as the output. Right? As that's the activation function there. So the things that need to be updated and learned to have an an effective perceptron is obviously these weights. These play a really critical role uh in determining what the output is going to be in any prediction. So, again, what does this look like? Um it's very much similar to linear regression. We talked about this earlier. And for that reason, this model only works really effectively when the data is basically what we would call linearly separable. So, if our classes, like let's say we just had two features, X1 and X2, um if we had, let's say, these red points belong to one class, and then these blue points, um or let me draw them in green, belong to a different class over here. If, you know, if we could draw a straight line through here and say everything on this side belongs to, yes, and everything on this side belongs to, no, those binary classes, then this model would be effective because it's essentially going to find that line and compare it to zero. Essentially, as if we shifted this line to be kind of at the origin, and we're checking like, does this result end up positive, meaning it's on this side of the line, or does it end up negative, meaning it's kind of on this side of the line, which would be a zero. Um so, the perceptron's really only good to be used because it's so simple, just a single neuron, it's really only good in these situations where the data is kind of what we would call linearly separable, which is not always the case, right? It's typically not the case, um which is why this model we need to expand this to have many neurons with non-linear activations to model like real-world problems, right? So again, the perceptron we get a zero or one. We're going to adjust the weights during the training process. Um They get updated kind of in a gradient descent style calculation that happens to adjust those according to the error. And the perceptron really only works well when the when the data is linearly separable as we just talked about. Okay, so it's everything here that we just talked about weighted sum through an activation. It's just this prediction is now we know what it is for a perceptron. We know that this is going to be a zero or a one. And we know what this activation is going to be. It's it's the you know, checking if that weighted sum is positive or negative or zero. Okay, so we know what these components are now for the perceptron. So everything we've talked about is is here. Nothing really changes except a particular activation for this binary case. And it's just a single neuron. So that's the other thing. We've been talking about a network of neurons. The perceptron is so basic it's just a single neuron. All right, just a single one. It doesn't have a layer of multiple neurons. It's just a single one. So very basic. So the components there are again what we've been talking about. They we have a set of inputs, we have a set of weights, we have a bias using for shifting which is important for checking relative to zero. We have a summation function that just takes the weighted sum and then we have an activation which is exactly what's producing our zero or one, right? The activation. Everything is there that we typically will have in any neuron and certainly in a network of neurons we're going to have lots of these guys. Okay, so the perceptron is a good model of a single neuron um in the sense that it takes a weighted sum, takes an activation, and produces an output. Now, what would happen with this that output in a in a larger network is we know that data would flow forward to the next layer, right? We know that um in a larger network, but in the perceptron, we really only just have this, right? We just have some weighted inputs going into that single neuron. Um but typically that data would flow forward in what we've said is forward propagation. Another word for this for uh type of network where data only flows forward in a forward propagation manner is um sometimes known as a feed-forward feed-forward network. So, so feed-forward is basically just a network of perceptron neurons, right? That that perceptron being um their computation is just a weighted sum through an activation, and it's just flowing forward. The output goes into the next layer, that output goes into the next layer, and on and on. So, this is sometimes known as just a feed-forward network because we do later on when we have RNNs, we'll have the ability to have a recurrent connection that goes backwards. But right now, you know, these especially with perceptron style calculations in these neurons, um everything here is just what we call feed-forward. Um everything just forward propagates. Nothing Nothing Nothing has a kind of feedback connection yet until we get to RNNs. So, this is another term you may hear is a is what we call a feed-forward network. It's really just um whenever you hear the word feed-forward network, you should think of perceptron style computations that just flow forward. Weighted sum through an activation going forward. That's all. Passing their output as the input to the next layer. So, some of the features of this is that information only goes one way. Um it goes straight, it goes forward propagated through, never touches the node twice. It has no memory of the input. Um which is what we would have in an RNN. We would have memory cuz we care about sequences, we care about data that's prior to our current um value that we're processing in the sequence. Um so it has So, the the feedforward network, which is the most basic type, that perceptron style computation, data just flowing forward, basically has no ordering, no sequential memory, nothing like that cuz it cannot go backwards. Just every all the data's um flowing forward. Okay. So, what we're going to do now is just look at the extension of the perceptron. So, given that we have a perceptron, remember the perceptron is just a single neuron that has inputs. So, we have inputs flowing into a single neuron, which is generating a single output. All right, that's perceptron. It's just this. It's not even a network really. It's just inputs uh >> [snorts] >> weighted sum of those going forward into an output. However, if we stack together multiple layers of these guys in terms of these neurons, we can generate what's called a multi-layer perceptron, which is is starting to be a very basic network. Right? It's not a full deep network in the sense there's not many, many layers, there's not many, many neurons. It's very limited. So, a multi-layer perceptron is designed to be pretty limited in terms of its structure. It's not a huge network. It's mainly just a small collection of maybe two, three, maybe four at most perceptron layers. Um and and we generate an output which could be one or two or three outputs, depends on what we're trying to predict, but it's Think of just stacking these neurons together in a simple collection of layers and we're starting to build a network that is the multi-layer perceptron. Um now, every neuron in the MLP, sometimes this is known as MLP, multi-layer perceptron. Um every layer in this multi-layer perceptron is just the simple weighted sum through an activation. So, this is still every single neuron still functions the same, weighted sum through the activation, right? It's It truly is that, weighted sum through an activation. So, nothing really changes there. Every neuron is like that. Okay, and all the data still feeds forward. It's just now we have multiple neurons and multiple layers, maybe two or three or four of them. Okay, so a perceptron um is decent except it fails at the separable uh things that are not linearly separable. So, a very basic example people like to give of where the neuron model of a perceptron really falls apart is like a simple um what's called an XOR, which is uh the like the exclusive or operation between uh um like two binary values, like minus one and one, minus one and minus one. Um if you take the exclusive or between the two, you can see what their values are. Like the exclusive or between minus one and minus one is still minus one. Um the exclusive or between minus one and one is one and then um what's interesting is you know this is not uh separable. You would have to in order to separate these two classes, you would actually need two lines or basically something non-linear. Uh because you would need um you would basically need to draw something uh kind of like this. Um in order to separate the two. Uh and say like everything on this side is uh belongs to one class and everything on this side belongs to another class. And that's highly non-linear. So and this is just one example of a function that would break the the you can think of many more like you can have like let's say um going back to the example we could draw circles here and here and then draw um the red [clears throat] ones um like this one could be on this side and this one could be on this side. And again, this makes it really difficult to draw a single line. Like you can't draw a single line through there and say well I can perfectly separate everything on this side and everything on this side. That's not going to be possible with with a function like this with data that's like this. Um so that's just to say the perceptron is not perfect. It's going to struggle on data that is not linearly separable which is what this means meaning you can draw a perfect line or a hyperplane and all the data's on one side of it and all the other classes are on the other side of it for a binary classification. Um you you most likely need a multi-layer perceptron because you need non-linearity. Right? You need this this kind of picture here in order to better separate it which is which is not linear. Okay? So that's just to say a single neuron by itself is kind of limited. That's where you need layers of neurons to more effectively model problems. That's all we're saying here. Okay, so I have a demo for you guys, and this is just going to use um so it turns out that perceptron a single neuron is actually available within scikit-learn. So, we don't even need to use TensorFlow yet or PyTorch yet in order to just work with perceptron. Um so there's actually just to show you that quickly, there is a a demo so this 3.04 demo inside of um lesson three here. Which let me go over to that and show you. So, we'll do this demo. There should be a couple in lesson three. We're going to do this 3.04. I'll give you a moment to pull it up. It should be in your uh lesson three materials. And there's also a I believe a data set that you're going to need, which is the spam the spam-based data set, which comes in the lesson three data sets. I can upload it here. But do you guys have the Do you have this notebook from the lesson three? You might need to download it from your reference materials. Okay, great. Okay. So, let's um let's practice building just a perceptron. Now again, it would we typically use a perceptron? No, we would typically use a a uh neural net that has more than just a perceptron, which is a single neuron. So, but the perceptron is an important building block to larger and deeper networks because it that fundamental idea of taking a weighted sum through an activation is really critical. And it's it shows up in a lot of networks. So, it it is an important building block, but by itself is not necessarily an effective model, especially in those cases we just saw where it's not linearly separable. Let's see how we use the perceptron here. Um what you can see is look at where it comes from. It's from scikit-learn in the linear model family. Now, we've seen other things from there, too, like the uh logistic regression, right? We've also seen linear regression there, but for classification, we've seen logistic regression from that linear model. Here is perceptron. It's considered to be in this linear model family because we're taking that weighted sum, which is linear, but then passing it through an activation, which is uh checking if it's positive or or negative, right? So, let's load Let's run that to do our imports. Okay. Let's load our data. So, we're going to load that spam-based data. I do need to put it in here. Okay. So, that's loaded, and we can check the uh head of it to see the first five rows, of course. So, you can see there's some uh some data. Now, this this spam is is whether or not the email was spam or not. So, it's got a It's all a bunch of data about the uh about the email. And then it has a classification category of spam, which is the last column, which is a one or a zero for not spam, right? So, that's This is a classification problem. Uh in that in this data is just a bunch of features about different words. So, for instance, this is a word frequency of the word all, the word address, the word make, the word hour, over, remove. So, it's got a bunch of word frequency features. And then it's got some capital capitalization features, characters like the frequency of the pound symbol, frequency of the dollar symbol, frequency of exclamation point, um parentheses. So, it's got a bunch of features about text uh within the uh email. Okay? All right. So, what we're going to do is um just have a simple check here if there are any nulls, we're just going to fill those with zeros. So, um obviously we could do a more robust check here. Um in fact, let's let's check uh you know, we know how to do data. dot is null and then dot sum, so we can see that across all the columns uh if we have any Um doesn't look like we have any. So, this Yeah, looks like we don't have any nulls. So, this should be relatively uh this shouldn't do anything essentially cuz this should fail. There are none, so this should be false. It shouldn't have to fill in anything. So, that that's fine. It's just going to produce our data. All right. And then this is the this is the important part this is the important part of grabbing our features and our labels, right? So, this is grabbing our features and our labels, right? Features and labels. So, this is grabbing every row and every column but the last one. That's what Iloc does, right? Do you guys remember that? Iloc. The for our data frame, this is grabbing every single row but then every column but the last. So, everything up until the last which is the minus one index. That is all of our features. And this is grabbing the very last column. All all rows, that's what the colon means, all rows, and then only the last column, which is that's the label. Right? This is the label the spam label uh column is the last. And we saw that in the in in this uh head of the data frame. Right? Spam is the last one. Every other column before it is is a feature. So, we are just separating those out, pretty typical. Separating those out, and then we're doing what we usually do with that is train test split, right? Pretty standard, where we pass in our features and our labels, and we do a train test split. Um and we can choose whatever test size. I think 0.4 is a little big. We could lower that to maybe 0.3 or 0.25 or 0.2. Um but I'll just leave it for that for now. Um but this is our usual train test split. So, nothing that interesting here, just generating our our training data and our testing data. It's pretty standard. Right? Nothing that uh out of the ordinary there for a supervised problem, which this certainly is, even though we're using perceptron, which is a neuron model to do it. Still supervised. Let's scale our data. So, we're going to scale all of our features. Um that's uh still something we're going to do here. Um using this perceptron uh scaling all of our features here. So, scaling our training features, scaling our test features. So, we train both of those, or sorry, use both of those. And then we can fit our perceptron. So, look how easy it is to initialize a perceptron um and and uh basically, the perceptron is um just a single neuron model, right? So, that's all it is. And it's going to use um uh it's going to train. So, this uh this.fit, I should call this out. this.fit is going to train the neuron which is just a perceptron, it's a single neuron. It's going to train this by learning the weights through uh gradient descent uh algorithm. So, it's going to run a gradient descent in the background to learn what those optimal weights are. Now, we haven't studied gradient descent in detail. We will as part of like when we get to the part on uh coming up shortly on learning how to train neural nets, we're going to learn a lot about this because every single neural net trains through gradient descent algorithm one way or the other. It does gradient descent. So, it's not going to be any different for the perceptron because it is just a neuron. It's a very basic It's not a network per se, but it is a neuron. The network is just a huge collection of neurons. So, um they're all going to train through gradient descent. So, we'll we'll learn more about that as we go, but this.fit is going to run gradient descent on our um our our features and our labels here for the training data, and it's going to the perceptron is going to adjust all of its weights um according to this data. >> [snorts] >> Okay, so we get a perceptron that has been fit. Um and then we can make predictions. So, we can pass in our training data to get a collection of predictions for our training set. And then, more importantly, we want to evaluate that on our test set. So, we pass in our test features to our model and do dot predict. So, so far this is just behaving like any other scikit-learn model, right? Nothing that interesting. But, it is cool that the perceptron exists inside of scikit-learn as kind of like any other model. Okay, so we can see the accuracy here on our data, and we get about an 89% accuracy on the training and about an 88% on the test. So, pretty decent. That's not too bad. Um using that perceptron. So, pretty cool. Um that works out, and that's just a That's just a simple neuron. It's not You know, this is a very basic model. Um it's just that single neuron. Of course, where we're going with all this is to build up our own neural network. That's going to be a huge collection of these neurons, right? That are going to be um organized into layers that are connected into other layers. And so on, and so on, and so on. So, we're going to build up to that. And one of the things we're going to have to learn is how to construct that in like PyTorch or TensorFlow. Um so, that's going to be more involved. And we're also going to learn how to set up this training algorithm, this gradient descent. We need to learn more about that. So, that'll be upcoming as well. Okay. And we covered this, but I just wanted to start here with kind of review of what we had talked about in terms of neural networks, right? So, just to refresh ourselves we're dealing with uh neural networks in general in um we know that those are comprised of kind of a network of neurons, right? And and um these are organized into layers, where every layer has a different number of neurons potentially that we could we could specify how many there's going to be. But, the the idea is that the data that we pass in will kind of flow forward through this network, and every neuron is really doing something like this. Remember, this is and this is the equation that we kind of had for a perceptron, which is that we take a weighted sum times the input. Um and so, this W star X is really meant to be kind of a dot product, like a weighted sum times inputs, right? So, we have weights times inputs. And then, we might add a bias in there as well, which is what the B is here, so adding in a bias, which is just another weight, really. And but, the important thing is passing that weighted sum of inputs through an activation, which is this F, right? So, we pass that through some type of activation, which generates the output for that neuron. And what happens in the network is these outputs just flow into the next layer. So, whatever is output from from this neuron becomes the input into this next neuron. It also becomes the input into this neuron and and this one. Really, all the neurons that are in the next layer. And of course, those have different weights associated to them, so there'll be a weight here, a weight here, a weight here that that dictates how much of that input from that neuron gets contributed to to this neuron's output. And same with this guy, like this guy's going to contribute a certain amount to this. This guy's going to contribute a certain amount to this. And there's going to be some type of weighted sum through an activation. And that goes from every layer to every layer. Right? So, um that's what we've kind of learned so far. The things that we need to study a little bit and we will coming up next are what are these activations in particular and when do we use certain activations? So in certain situations, you know, a certain activation will make more sense because they can if you look at the activation from a math perspective, they really control what the output of this is allowed to be from a range perspective, right? Because you're taking us an input passing it through this function and generating an output. So depending on what that activation function is kind of controls what you can possibly get out of there. If you're predicting things like in a in a probability sense, you probably want some type of activation that's going to limit your range to a probability like between zero to one, that makes sense. If you're doing a regression, you don't want to do that. Usually want it to be just any real number value, right? To to mimic like a price or temperature or something like that you're doing a regression with. So this activation is really important and we're going to talk about some very common activation functions that are out there. But this is what we're talking about with neurons, right? Weighted sum through an activation and we're doing this in this network of neurons that are kind of spread out amongst these layers passing data forward throughout and you finally get to some output and even the output, remember the output when we get into larger and larger complex use cases, the output is going to be different depending on what kind of problem we're solving. So if we're just predicting like a binary output, that's going to be just kind of a single output that's a probability, right? Between zero to one. Kind of like what we saw for logistic regression for like a, you know, spam, not spam, fraud, not fraud kind of use case. But if we're predicting, maybe we we're predicting between three different kind of image categories like this is an image of a cat, this is a dog, this is a giraffe. Um we would we would likely have three outputs, right? We'd have probability of dog, probability of cat, and probability of giraffe. It it or you know, if we had 10, then we'd have 10 outputs here. Um so our output layer should kind of match how many outputs we're trying to produce. Um that makes sense. And so we're going to see that with different use cases. When I set up those output layers to contain the right number of outputs and also, you know, actually be probabilities, which is going to be hugely dictated by this uh activation. Right? We want the activation to generate probabilities for those neurons. Okay, so this is a little bit of a recap. And if you guys remember um perceptron is a a special type of uh neuron it that is just a single neuron that has an activation function that produces a zero or one. Right? It's It's like a step function that produces a zero or one um depending on if this weighted sum is positive, this is going to produce a one. If it's negative, it'll produce uh a zero. So that's what we meant by the perceptron model. It's just a single neuron that does this activation. Hopefully that rings a bell from last week. And And what we're really interested in doing with bigger and bigger neural networks is basically having many perceptrons, which are basically all of these models of neurons here. So you have lots of perceptron style uh calculations, and that's where we we graduate from just a single neuron to multiple layers of perceptrons, which is which is this MLP model, multi-layer perceptron model here. So that's that's kind of where we're headed, and then this is going to keep expanding. So we're going to have even more layers, even more neurons, and then eventually, even more complex computations there that are not just weighted sum through an activation, but things like convolution, things like recurrent layers. So, we're going to work our way up there in the future, but that's kind of where that's where we're headed with all this. So, just wanted to recap that a little bit um and pick it up from there, especially the the activations. I think that will uh we'll talk about that next. So, we we ended on this demo last time, which was um kind of going into scikit-learn. If you remember, we did this demo. Went into scikit-learn it and scikit-learn actually has the basic perceptron um model. Um but, of course, when we build a more complicated neural network, it's going to be in TensorFlow or uh PyTorch, but um scikit-learn itself does have um a perceptron um because it's it's a very basic model of a neuron, um but it's not really a network, right? It's not like a full network. It's just kind of a single neuron. Okay. So, before we go into activations, where I really want to pick it up here um and talk more about uh what actually happens with these activations and what are some good examples of activations. Let's talk about activations then. So, uh we know what the definition is uh in in terms of it is a function that we're going to apply to that uh inputs that are generated for the neuron where we have that weighted sum uh with weights times inputs, and that generates this value that we're going to pass through this function, calling that function the activation function, which gets the name activation function. It really comes from biology and and kind of the actual um workings of a neuron, which uh gets activated if there's enough kind of energy there. So, same kind of thing here. If if the input is enough, we run that through a function, and we kind of generate this uh output of it through this activation function. And so, that's where it gets kind of its name. Now, in terms of the perceptron, we know what the activation function is. It's It's actually um a very simple function which looks at looks at this weighted sum of inputs. So, it takes this um weighted sum. So, you can see here's like an input one, input two, all the way up to input n. We weight those. So, there's a corresponding weight for every one of those inputs. We know that. We total that up and maybe even add in a bias. So, you can think of an extra term here. There's a bias that it get uh contributes to the sum typically. And then this goes through this F, right? This f of x um activation, which is taking the f of that uh weighted sum uh that is all of these guys, right? So, it this activation function is being applied to those. And really in the perceptron, it's a simple function. The f really says that if So, that the f um is such that um it has a definition such that like if um this weighted sum is positive, uh then we produce um if this sum is positive, then we produce uh one. Um else, we produce uh zero. Else being, you know, it's negative or zero, then we produce zero, right? So, the activation for the perceptron is really, really simple. But that that's not the only activation that's out there. And in fact, when we start building our own neural networks, um there's going to be many different activations we can choose from. Some of them are going to be more um popular than others. And some of them have very particular use cases. As I said, one thing you want to pay attention to is this activation really helping us kind of narrow down the range of what the output of that neuron could possibly be. So, like in the perceptron case, the output's only allowed to be a binary one or a zero, which is good for certain problems, right? It's good for like a spam, not spam, cancer, not cancer. But, it's not very good for regression at all, right? It wouldn't be useful at all for regression, wouldn't be useful at all for generating a probability even cuz it's still structured to just zero or one. Uh and it certainly wouldn't be good for multi-class where we have maybe 10 different categories of things we're predicting. Um so so the perceptron activation is very limited. Um but, we're going to explore some other ones that are found in a lot of a lot more general neural networks than just a perceptron. But, but you see what its purpose is, really to produce the output of a neuron um passing in those inputs through this activation function. So, it has that capability of kind of um mapping those to a particular range, which is important in terms of what are we trying to get out of the neural net, especially as you get towards the the output layers, right? Then you really care about what your final output is. Okay. So, if you think about it, if like what would happen if we did not have an activation function? So, just think about that for a second. If we didn't have that, um then in theory, like you know, depending on what those inputs are, we really could get anything from this weighted sum with with the weights and inputs, right? They could be any real number. That's all that it's saying is like you get kind of an infinite range of possibilities. It's not really limited to anything. Um which is fine for some problems. Like a regression, that's okay. But, that is very open-ended and not very good for other problems. Like Like if we want to generate a probability, this is really bad because we don't want things to be really large or really big. We actually want it to be confined to be between zero zero or one, right? Coming out of the kind of final neuron. So, having an activation helps us kind of bound that output coming out of a neuron because if you think about it, this weighted sum could be anything. And actually, even in a regression, it's probably not good that it could be anything negative or positive. Like if you're predicting a price, you probably want that to be positive. So, even minus infinity to infinity is probably not even good enough. You probably want it to be restricted to between zero and infinity at the very least. So, that's where the activation comes in. We apply a function to this and that helps us kind of map that to a potential range. Like maybe we map it to zero to infinity. Um so, we So, for like a regression, that'd probably be good uh for things like price, temperature, whatever. Um or maybe we map that to uh zero to one for like a probability. Um that that would be good for that case. So, it just depends. It just depends on what problem we're solving. It also depends where we are in the network. Usually for things towards the end, we want to make sure we're using an activation that lines up the output to be in the appropriate range. Like if we're producing probabilities, want that to be between zero and one, not, you know, all over the place like negatives, positives, which just the weighted sum times inputs could be anything. Right? Could be anything. Um and again, that's where the activation function kind of comes into play. It's applied to this and allows us to kind of map this uh to get a certain output from the output range at least for that neuron. Okay. So, if that makes sense. So, for that reason, activations play a really critical role in in any neural network. And as I said, especially towards the end of the network, you know, it I the example I always go back to is if we're predicting like three different things like um dog, cat, and giraffe, we don't want those numbers to be all over the place. We really want them to be probabilities that add up to one, actually. So, across this kind of outlets, imagine this was our output layer, we would want to make sure that these are all probabilities that are between like uh zero So, probabilities in zero to one, in this range. So, we'd certainly want that to happen. We'd also want all of these to total up to one. We'd want them to be normalized in some sense, right? So, um each one of these outputs should um total across the output should equal one, right? That should happen if they're if they're probabilities. So, that's something we're going to be interested in doing it. And what we would hope is there's an activation we can apply across this entire layer, really, such that um all of these uh are probabilities, certainly in zero one, and total up to one for those kind of cases. So, we're going to see activations that actually will do that for us. We'll be able to normalize that for us, which is good. All right. Okay. So, and the other thing about the um activation function as well is the the activation function actually also serves a really important part of the model um because it allow it basically allows us to introduce um non-linearity. Because if you think about it, like if we just had Let me go back a couple slides to this. If we just had this, this is basically a linear model, right? Just like this. This is a linear model, which we've seen before. It's just a linear This is just a linear regression. Linear regression, right? That's actually all that that is. It's just weights times inputs plus a bias. We've seen that before, right? That's all that that is. So, without activations, we essentially have just a huge collection of linear regressions that are um connected together. So, activations actually play a really important role in introducing nonlinearity, which is really, really important for a neural network to be able to learn complex patterns. So, activations are actually really critical. So, it's having this F here, this activation, allows us to learn more general patterns than just linear ones. Cuz if we didn't have that, that's all that we would have is just linear relationships all over the place, right? Throughout the network. Um but, having an activation function here that in the activation function will generally be non-linear. It will generally be non-linear. We're going to get into some examples, but like some of them are going to be uh like the sigmoid, like from logistic regression, like a logistic function. Some of them will be hyperbolic tangent is a popular one. These are all, you know, highly non-linear functions. There's also exponentials that are possible. So, different types of functions that are all non-linear. That's actually a really big key, uh because we want to introduce that in the learning process is the ability to kind of learn non-linear relationships. So, that's another reason why they're important. They limit our range, but they also introduce nonlinearity so that our network can learn those non-linear patterns. It's really important. Okay. All right. So, let's talk about some of the the popular activations. Now, by no means is this going to be um an ex- exhaustive list. There's definitely more than what's just on this list, but I'm going to go through these cuz these are going to be the most popular that we deal with in this course, for sure. You know, there are other ones that are out there that are more advanced, um but I would say they rarely show up for the use cases that we care about. The ones The ones that we care about are mostly going to be what's on this list, um and I'm going to explain what all of these are and kind of what they look like. The thing that these all have in common, by the way, is they are all non-linear functions, all of them. So, that's really That's a really important attribute of an activation is generally non-linear. And again, we're going to have these spread out, you know, across all the neurons. And um so in that way, we're kind of like stitching together all these non-linear relationships, which is which is um how we learn complex patterns. But, let me go through these. So, these are some popular ones. We've actually already looked at this one a little bit, but let me I'm going to go We're going to start with that one, and then we'll talk about these other three, the sigmoid, ReLU, and softmax. We'll talk about all of those. So, these are some popular activations. We've actually already looked at step. That's the one that's involved in the perceptron. Um let's start there. So, the The reason this is called a step function is because it essentially only It has one step up from a zero to a one. So, if you look at it on a graph, which I'll do on the next slide, it's a function that um goes from zero up to one, and it's just a single step up uh one unit from zero to one. Now, this activation is kind of special to the perceptron algorithm because um that's just the way it's designed, right? Is that uh whenever that weighted sum of inputs that is being activated, right? That That data that's that weighted sum going into this function, Whenever that is, um positive, um we get a one as our result. Um otherwise, we get a zero whenever that weighted sum is negative or zero. So, we we get zero. So, in that way, it's it's uh let me show you. It's kind of a step function that looks like this. So, we This is inspired by activating a neuron, right? So, if you have a single neuron, it gets kind of activated when there's enough energy. In other words, or it's bigger than zero. And this is why it's called a step function is you can see on the graph, it literally looks like a step. Like like as on the stairs, right? It goes up and then over. So, what this graph represents is, you know, this axis is the input. And the Y axis is the output, of course. And what you're looking at is for anytime the input is negative, so you're on this side of the X axis, um you get zero as your output, right? You're you're down here. You always get zero no matter what that is. The moment you have something positive, meaning you're on this side, um you get a one, right? You get a one for all those, no matter no matter what that is. As soon as it's positive, you get a one. So, you get that step up to one. Um and that that neuron is essentially activated, right? And we get a one as the as a result. Um so so this activation function kind of looks like this. Now, that is non-linear, for sure. Um but what we also see is the range of this is very limited to zero to one. So, the this step function activation is really only useful for this perceptron when we are doing a binary classification that's going to be zero or a one. Right? Really only useful for that because we're not able to produce any other value in between, right? We only get a zero or a one. We also like for regression, we'd be out of luck cuz we can't produce anything else. Um so, definitely wouldn't be a good fit for any of those kind of problems, right? So, that's the step function. Again, this this one's only going to be used for the traditional perceptron. That's the only time it's really ever used is in that case. So, let's talk about the sigmoid. Now, this is one that we have seen before, and it's actually the same function that we have seen with logistic regression. Um exactly the same function with logistic regression. So, the sigmoid produces um a probability between zero to one, which also makes it useful for binary classifications. And just like the step function, this is helpful for binary classification. The difference is the sigmoid um allows us to produce a probability that could be anything actually between zero to one, right? So, in that way, we're actually generating a more direct probability rather than just a uh just either a zero or a one final result, which we know from logistic regression, we did the same thing, right? We produced a probability, and what we did is we said, "Okay, if that probability is big enough, meaning it's above .5, we categorize that as a one. If it was down here, we categorized it as a zero." So, we did that, but the sigmoid activation allows us to produce a zero uh a probability between zero and one at the output of a neuron. So, remember, we're still applying this function to a weighted sum of inputs plus a bias, maybe. And um now, this function is the sigmoid. Is what we're saying is called the sigmoid. Um and this is going to map whatever this is um to between zero to one. Some some probability. We interpret that as kind of a probability between zero to one. So, where is this useful? This activation function is mostly used for binary classification as the output neuron. So, if you're in the network and so let's say let's draw the network. So, let's say we have this network and we have maybe we have a hidden layer here and then we kind of produce one final output. What we would if we're doing binary classification like spam not spam or cancer not cancer, what we would really like this activation to be is a sigmoid. This this guy should probably be a sigmoid because what what we want to do what we want to do is basically produce a probability from this weighted sum of these guys coming into this, right? So, we're going to take this weighted sum and generate a probability out of that. So, we want this final result to be a sigmoid. Now, these these others in the network don't have to be sigmoid activation to produce their output. That's fine. We're going to look at other activations that are more typical for kind of the middle of the network that produces a general value, but sigmoid is really really often seen at the very end when you want to produce a zero or one. Generally, you don't want to produce a zero or one kind of in the middle cuz that limits your range, right? That limits cuz like then this this neuron would be very limited to a probability. This neuron would be kind of limited. This would be limited. We don't really want to do that in the hidden layers of the network. We don't want to limit ourselves cuz then we're not going to learn very effectively. But, as a final output for a binary classification problem, yes, we want that to be a probability between zero and one. So, sigmoid works great. Remember the the function for a sigmoid is what we saw in logistic logistic regression. It's this um fraction that looks like this. So, it's it's uh definitely non-linear, right? Definitely non-linear. You can see it from the graph. But, certainly, you know, this produces uh this produces um values between zero and one. When X gets really big negative, um this fraction in the bottom kind of grows exponentially big when X goes this way. It grows the the fraction grows exponentially big in the bottom and makes it one over something really huge, um which is basically zero. So, that's why it decreases down this way as the X gets more and more negative. And then kind of the other way around is like if X gets really big positive, and you go this way, um you get closer to one because this term um basically decreases down to zero um the larger X gets. So, you get one over one. So, you get something closer to one. So, hopefully that graph makes sense like based on that function. But, that's that's what the activation is. And and again, this is the same that we This is some This is called the sigmoid. Sometimes you'll hear it called the logistic function. Logistic function. Um it's the again, it's the same one that's kind of used in logistic regression. Uh in logistic regression, remember this it's basically this, but we have um we have uh we have something like this beta. We had our beta I X I. We were doing that. Um so, we essentially have that same thing uh for this activating that that neuron. Okay, any questions about the sigmoid? So, when you use it is generally the last neuron in a binary classification, if you're building a network for binary classification, it should be the activation on the final neuron. That's what it should be to produce a probability between zero and one. We generally won't use it in the interior of like the earlier neurons because we don't want to limit the output of those earlier neurons prematurely, right? That that will that will basically collapse the learning and it won't be as it won't be as effective. But we do like if we're doing binary classification, we do want to produce a probability as kind of that final output of the network. So that's why you'll usually see it as a last activation here. Okay. So sigmoid will sometimes see. Now let me show you the most popular uh activation and the reason this is the most popular is because it's the one that's used almost exclusively in the hidden layers of a network. The reason this is because it does not really restrict the values. In fact, what this this uh activation will just basically return back to you the input assuming that it's positive. So as long as it's positive, it's just going to give you back what you put in. So see how this function gives you the maximum between zero and X. So what that really means is like as long as X is positive, what's the max going to be between zero and anything positive? It's going to be whatever that other what that positive number is, right? So this will always just return to you what you put in as long as it's positive. However, this is a non-linear function because it has this piece right here, which is really critical. This piece is saying like whenever we have something negative, it's actually going to chop that off and just give us zero. So this is actually discouraging negative outputs. It's always going to give us something that's either zero or positive. Right? So, this this activation function is known as a rectified linear unit um because this y equals x is a perfect linear line um with a slope of one. But, we're rectifying that with um this kind of non-linear piece, which is this uh flattening of anything negative to zero. Um because a true line would continue this way for anything negative, it would still it would just That would be the true line, right? Would be um anything negative would just give us that value back in return, but we're actually chopping that off in favor of making it just zero. So, rectify it that way. The way you pronounce this is called ReLU. So, ReLU, r e l u. Usually how you'll hear that pronounced, ReLU. This This is This activation is the one that's most often used in the interior of that network, those hidden layers. Um because it allows us to basically generate the outputs that we care [snorts] about um as long as they're positive. So, it it doesn't limit us down to a specific range like zero or one. It does limit us to a bit wider of a range than that like zero to infinity essentially, but in this case like these guys would likely be all ReLU activated. Um and then like this one would probably be a sigmoid for uh a binary classification. Doesn't choosing the max of the range skew the model in terms of prediction? We're not really choosing the max. Uh We're choosing the minimum uh because nothing can be negative, so it's always going to be capped at zero. Yeah. It So, in theory, yes. I think that's a really good question. But, in reality, no, because we're going to have many of these neurons um typically in the hundreds of these per layer. And basically, what we're going to do is adjust the weights enough to make up for that. So, remember, there's a bunch of weights here, and we're just going to make enough weight adjustments so that we can always produce something that carries forward into the next layer. So, um the weights Hopefully, that makes sense. Like, the weights really help us learn because we're going to adjust those weights accordingly to produce the right outputs. That's actually all of what training a neural network is is making the right weight adjustments. Um and that's what we're going to talk about coming up soon. So, not really. We just have to adjust the weights appropriately uh to to generate the right outputs. And that's something that gets learned during the training process. So, we'll talk about that. Okay? But, the this ReLU is um probably the most popular activation. We're going to use it quite a bit. It's used all over the place uh in most networks. Um ReLU is uh very very popular activation. It's It's kind of a very simple non-linear function, if you think about it. It's because it's almost always linear except in this case. This makes it non-linear, right? Is this piece of it kind of makes it non-linear. But, otherwise, it's mostly linear. Now, what a good thing about this activation as well is um like, if we were producing a regression value for like a price, this would be a good one to have at the end, right? Cuz you can generate anything that's not negative. So, this would be a good regression candidate to use as an activation sort of towards the end, right? To to basically make sure that you're getting something that's positive. So that that would be like realistic. Okay. So this is a popular activation. We're going to use it quite a bit and of course like in our code we'll be able to use this as one of our activations. And like I said, where you'll mostly when you would use this one is pretty much in any hidden layer, your default is going to be ReLU. There There's very few reasons you should not use a ReLU in your hidden layers unless you have a good reason not to. Um most like 90% of the time you're going to see ReLU used as as a activation on the hidden layers. The output layer is where you'll see those differences. You'll you can see a sigmoid. You could see that step function in the perceptron. And then we'll talk about the next one which is softmax which will which will be used for probabilities as well. But most of the time we'll see ReLU show up in those hidden layers um between the input and the output. ReLU is very very popular. It's just an It's a very effective activation. It's a very effective non-linear function that kind of just keeps things moving forward um as long as they're positive. Okay, so what's the difference between the sigmoid and the ReLU that we've looked at so far? Well, the sigmoid has a uh it's a little bit more complex of a function. It's that exponential that logistic. And one of the issues with the sigmoid is it can be susceptible to what's called a vanishing gradient which we'll talk about later. Essentially, this is this is a a consequence of the sigmoid always being between zero to one. So the issue with that is it's a fraction. Right? A sigmoid is going to produce a fraction that's between zero to one. The issue with that is when we're doing the gradient descent learning process, if we're carrying around a lot of fractions, this is why you'll never see sigmoids in those interior layers is because if you're carrying a lot of a lot of fractions, they they multiply together with the because they become the inputs to the next layer, they multiply together with the weights of the next layer and really decrease that and and really limit your learning ability because you have such small fractional values. Whereas ReLU can avoid that because it is generally just going to give you what you put in. So, it's going to have a wider range, right? It's not going to limit you to 0 1. So, it but it's still non-linear. So, so it can help you learn those kind of complex relationships still with the help of kind of the weights in the training process. So, this is just to say you should generally be using a sigmoid in your output if you're producing a a binary classification output. You should avoid using sigmoids kind of in the hidden layers if you can cuz it's going to make learning more challenging. It's going to make training the neural net more challenging and we'll also study a little bit more the details of why that's the case, but it's mostly due to its range. It's producing the sigmoid produces fractions, which carry through that throughout the network, and it it kind of compounds. Like if everything is a fraction, things are going to compound really quickly and and make small values that are very difficult to learn from. Okay. So, that's why you typically see ReLU in those uh hidden layers um because it makes sense. You don't want to have these small fractions kind of all over the place. Does that make sense? Like if I go I Actually, let me just illustrate that real quick cuz like you can imagine if we had uh going into a neuron, if all of these guys were fractions coming out of a sigmoid from from other neurons in the layer, um that's just going to compound, right? Like if this is if this is 1/3 and then this is like uh 1/2 and then you have um maybe another neuron that's connecting to it that has like a 1/4, then you just have a bunch of fractions that are being weighted. Um and they're they're always going to reduce these values and and limit your range and kind of compound things. Um so so you don't usually see those sigmoids in the hidden layers. You'll see them at the end if you want this to particularly be a fraction. You'll see that. Okay. So ReLU, non-linear, it's just the maximum between 0 and X, whatever's going into that activation, and you'll typically see these in the hidden layers in between the input and output. All right, I want to cover one more activation because it's also going to be really popular especially it's really popular especially as the output using it in the output layer for multi-class classification. So the softmax actually just extends the sigmoid. It's like a multi-dimensional sigmoid essentially. So the softmax produces values between 0 and 1 just like the sigmoid does. It produces probabilities. The difference is that the softmax is useful for generating multiple probabilities that add up to one. So what you'll typically see the softmax used for is the activation across the final layer in the output if it has multiple neurons in that final layer. So let's say let's say we had three. So we we have a neural network that kind of looks like this. And then we just have um maybe a hidden layer of four and then our final output is three because we're we're predicting um dog, cat, and giraffe, right? Those three animals. Um what we would what we would use for this activation in this layer is the softmax. We would use the softmax for each one of these. And what that's going to do is produce probabilities that such that all of these are between zero to one and they actually are normalized such that they total up to one. Um so softmax is really, really great as you'll see it in kind of your final layer if you're if you're doing multi-class classification, right? So you so you're predicting amongst a thousand things, then you'll have a thousand probabilities that total up to one. And in this case you you we only have three. So you have three probabilities all that are between bounded between zero and one that total up to one. Um so you'll typically see this in the output layer um especially for like image classification, um you'll see that or it could be, you know, multi-class classification. Uh you'll see you'll see this softmax. Another place you see softmax is in um like a an LLM, uh like a a large language model will use the softmax as its final layer because it needs to produce probabilities across every token that's that's possible, right? So and the reason is is because the model like a GPT is going to predict what is the most likely next token and so that'll be represented by one of these probabilities, right? What's the highest probability? As kind of the most likely next token. So softmax, incredibly useful for anything that's multi-class classification and it should be your last it should be in the output layer, right? It's going to be the activation that's used on the output neurons in the final layer to normalize them between zero and one and and get them to add up to one more importantly. So, um you know, it's a it's a really uh it's a really critical function. The softmax function, if you're curious what it is mathematically, is sort of like the sigmoid except it's um more of uh it's more of an exponential. So, it's kind of like a process summation of these. Um so, you uh you actually apply the softmax to individual components and then you sum that up over all the components. Um And so, this normalizes it to be between zero and one and uh make sure that the total of it is one um effectively. So, it's a function that looks kind of like this. And you apply it to a component in the layer. So, every neuron in this layer is going to get uh this function applied to it. Okay. So, so just to recap that, sigmoid it should be your activation on your final neuron in a binary classification. Um ReLU will be in your hidden layers in the middle. It's the maximum between zero and X. Usually just returns X as long as it's positive. And then softmax is going to be in your final layer for multi-class classification to produce probabilities that total up to one. And by the way, these are all all these activations are built into TensorFlow and PyTorch. So, we'll be able to use them really easily inside of our code. Um they'll be built into those libraries. So, uh it'll be really straightforward when we declare a layer of neurons, what activations do we want to use? Um it'll be really straightforward to set that all up when we start building our own networks. So, all of that will be readily available and there's really good documentation in both of those libraries on kind of what activations are available. Like I said, there's more than this. You know, there are other ones like some pop some popular ones are like hyperbolic tangent is a popular one. Uh especially with some image problems, you'll see this tanh be used hyperbolic tangent. Um that's a popular one. Um there's other there's variations on uh on ReLU. There's there's one called GELU. Um there's there's one called leaky ReLU. There's all There's actually a lot of them, but um the ones that we've covered on this list are the ones that we'll primarily stick to uh as we go through our examples. Okay. So, that's a little bit about activations. So, just saying that there are, you know, definitely multiple of them and uh which one we use depends on really the situation. All right. So, wanted to do a little bit of a demo here to show you an example with some activations uh and kind of how the activation is applied. Let me jump over to that. Okay. So, we're going to look at this notebook. Do you guys have this one? This is the 3.06 uh taking a look at neural networks and activations. Hopefully, you have this one you want to follow along. Okay. Great. Okay. So, what we're going to do the purpose of um Thank you for Yeah, thank you for sharing that. The purpose of doing this demo is uh mainly to see how we can build essentially a simple um neural network from scratch. So, not using TensorFlow or PyTorch, but see how we can use like a weighted sum through an activation and see that actually work in code, which I think will be really interesting to see. Um and and really start to put the pieces together of a what a neural network looks like. Um so, you can kind of get a feel for the different components of it without yet going into our uh frameworks that are going to handle that more in depth like TensorFlow and PyTorch. So, let's see. So, first of all, we're going to import some things from NumPy. So, some functions we might need like the dot product, um random value initialization, um because we're basically we're going to build our own neural net that has those weights. So, we're going to start out with random weights. We need some randomization here and uh we'll have some various like exponential function and dot product, things that we may need uh from NumPy to do this. Okay, so what we're going to do is um essentially we're going to start by building a neural network, uh a fake neural network. I say fake because, you know, it's not going to use our frameworks that we're going to learn about, but it's going to show you the the basically from scratch some of the components that you would see in a typical neural network including the activation, which is really important. And what we're going to do is basically build a perceptron neural network. So, it's going to it's going to take uh uh three inputs essentially um an input that's a size three vector, basically a three by one matrix, and uh it's going to map it's going to use this particular activation. We're actually going to use the hyperbolic tangent activation and and produce an output um which which would be kind of like a regression almost, but just to show you the different components of this network to you so you can see what's really involved with it. Okay, so to show you what's going on with this fake network, so we're going to build our kind of network from the ground up, which is going to be kind of like a perceptron. What you're going to see is we need some weights. And we have uh these weights which are generated randomly um and this ensures that their range is between minus one and one because this uh numpy.random.random generates a random value between zero to one. Okay, so if we multiply something between zero to one and multiply that by two uh and subtract one, kind of the range that we limit ourselves to is between minus one to one in terms of those weights. So, these are just going to be randomized weights that that live somewhere between minus one to one. They're going to be random though. And the reason we're doing that is just to start out with randomized weights. This is actually how a network usually works is when you initialize the network, it has random weights to start with. What needs to happen is you have to train your network so it learns those weights during the course of the training process. But this is pretty standard. Uh this is pretty standard that this is going to be randomized to begin with and when we start building our own networks in in our frameworks like PyTorch, TensorFlow, um those will also be random to kind of start with and then they get trained, right? How is this plural? Um because there's three of them. This random uh 3 1 generates kind of a uh a vector of size three. Um technically it's a matrix that's three rows and one column. So, there's multiple weights. Yeah, that's that's why it's synaptic weights um because there's three of them that are just kind of in this like 3 by 1 matrix, which is just basically a vector, right? With with three weights. Okay. All right, so then we have our uh activation function here, which is going to be hyperbolic tangent. Now, all we're doing is when we give an input to this hyperbolic tangent, we're just going to um apply the numpy hyperbolic tangent to that input. Now, what this is actually going to be our activation. Now, we could use any activation we want. This is just an example. We could use ReLU. We could use sigmoid. This is just an example of declaring an activation and using it in our network. So, I'm going to show you what that looks like. But, you know, if it was ReLU, then that would be you know, it'd be like max between zero and X. Would be the ReLU. But, in this case, we're doing np tanh, which is hyperbolic tangent. Applied to X. Okay, that's just a function. Just an activation function. Okay. Now, the other thing we have here, which is I'm not too concerned with right now cuz we haven't learned we're going to in in the later notes in this lesson. We haven't quite learned it is the gradient descent process for training a neural network. We're going to learn all about that coming up shortly. But, one of the things you need in order to in order to do gradient descent is the gradient of the activation. Which is this. So, the derivative or the gradient of the hyperbolic tangent is this function here. That's just a mathematical function that involves that hyperbolic tangent. So, the reason we need this is the reason we need this is for training. Gradient descent. And again, I'm going to go into detail on what gradient descent is coming up shortly. In the notes. But, it's just here in this example for the an example of how you would train this thing. Okay, so we have the activation derivative, which will be useful in the training for gradient descent. Okay, and then we have a function called train. So, this function is called train and takes the training inputs, training outputs, which are kind of like the labels, and then a number of training iterations. Again, this here is going to be our this is going to be gradient descent training process, which we haven't learned about yet, so I'm not too concerned with us digesting all of this this moment. But just so we kind of preview what's to come, essentially what we're doing here is an iterative process through our through through our data. So, we basically iterate a certain number of times, which is configurable, this number of training iterations. We basically do this we iterate this many number of times through this descent process, which is to say we basically produce the output given our inputs. So, we do a forward propagation. So, that's what this is is we do a forward propagation to produce outputs. And why do we do that? It's because we want to see how far off our network currently is. So, if we produce outputs, what we can do is actually compare that to our labels, which are these guys, and we can compare our output to that and see how far off we truly are. Um which is what's called our error or sometimes called the loss. So, this part here is computing the loss, which is how bad our network is currently. So, we're we're just gathering up how bad we are currently. Now, why do we do that? It's because we use that quantity to make an adjustment to our weights. That's the key thing is we actually want to make an adjustment to our weights to basically shift our network to perform better. Our goal is to minimize this error. I should I should write that down. The goal is to minimize this error. So, essentially what we're doing is measuring our current error and then using that quantity along with the derivative of the activation, which is here, the derivative of the activation, we basically use that amount multiplied by that derivative. We basically use this amount to figure out how much we should adjust our weights. So, this is a numerical calculation that's driven by this gradient. So, that that's why it's called gradient descent. Is we we essentially want to make an adjustment that will mini- that will lower our error over the course of the iteration. So, as we iterate a certain number of times, we should be getting better and better and better at making predictions using our network. And you can see here like we make adjustments. So, you see how we take our weight, we take whatever our weights currently are and we add in an adjustment. So, this makes a tweak. It Think of it like turning a knob. So, we essentially turn the knob and the adjustment is telling us how much should we turn that knob? Should we turn it a lot? Should we turn it a little? We're figuring out how much we should make that adjustment to get better outputs of the network. So, this is turning the knob to so the network can predict better. Right? So, it So, it can it can make the right adjustment and predict better in the future. So, this in a nutshell is how any neural network's going to train. Now, this process will generally be handled by the framework. So, by PyTorch or TensorFlow will manage this for us because it turns out when you have a pretty big network that computing this adjustment is incredibly intensive on a big network. Computing what that adjustment should be is is actually very computationally intensive and it's called backpropagation. So, So, basically uh doing the weight adjustment and uh figuring out uh figuring out how much you should adjust and then actually turning all those knobs you can imagine like for a very large network that's going to be an expensive operation. Just to give you some context like modern LLMs uh have hundreds of billions of these guys. Hundreds of billions of weights. So, you need to compute an adjustment for every single one of them. Hundreds of billions of weights, right? And you need And you actually need to do that every iteration. So, you're Imagine training for thousands of iterations and you have hundreds of billions of weights that you need to adjust. It's a That's why they take forever to train. That's They're massive networks. They take forever. That's That's why. Uh because of this computation right here can be very expensive uh because this derivative can be very uh computationally intense to compute. It depends on your activation. It depends on how big your network is, but generally those large networks like an LLM uh it's going to take a while. That's where you benefit from a GPU, by the way. It's to try to make this computation faster. So, we'll actually see that like when we build our computer vision networks, uh we'll do a similar training process and we'll utilize a GPU there to make this faster. Okay. By the way, I forgot to mention uh this forward propagation when we produce the output, uh we call it think, mainly because we are uh it's like sending data through our network, so it's like all of our neurons are thinking together in this network. Um so, thinking What is thinking? It's really um this is uh doing our weighted sum through an activation. That That's all it is. So, it's like our our weighted sum through an activation. That's all it's doing, right? A weighted sum through an activation. So, you can even see that here. Notice how we apply the activation, the tanh, to this weighted sum. Now, why is it a weighted sum? It's cuz you're taking the dot product between your inputs and your weights. So, that literally Now, what does dot product do? Remember, dot product adds together the product of these guys as vectors, right? So, that's that's taking our weighted sum and then Does that make sense? This is taking our weighted sum and passing it through this activation. So, that's This think is just computing that neuron essentially, right? Computing that neuron. All right, so just to finish out the demo, um that's the network. So, that's our That's kind of our fake network. It's got three weights. Um it it does uh it it thinks by doing a weighted sum of those inputs through So, it has three inputs, three weights. Take that weighted sum, pass it through a tanh activation. Um and again, the activation's kind of arbitrary. We could really have it be whatever we wanted. We could It could be a sigmoid if we're producing a probability. Here, it's just producing a tanh, which is going to be Tanh has its um outputs are actually between minus one and one. So, we're always going to get something between there. So, then we can run this code here, which is going to take a random set of inputs um and then the labels for each one of those. So, these are all like different labels. Like, so we have four different input sequences. Notice how every input is size three, which is what it should be. So, here's here's one input. Here's one input. Here's one input. Here's one input. These are more or less random, just to just to show you how it works. And then here's a label for each one of these. So, this guy is labeled zero. This This guy's labeled one. This guy's labeled one. And this guy's labeled zero. So, we have four labels here corresponding to each one of these sets of three inputs. So, here's our random weights to start with is this vector. So, So, we're printing that out just so we see what our random weights are. Now, when we go through the training process, which is going to have 10,000 steps and use those random inputs and outputs, look at what the weights are after we do this. The weights are now completely adjusted. And we can actually put in a new input and see what the output would be. The output, which is just running think on this input, produces a label of one. Okay, so the network functions as we would expect. Um, and the big thing is to to take away here is to see how those weights actually get adjusted. You see how far off they are from what they started as. So, here they started as these really random numbers between minus one and one. And uh, here they get updated through the training process. So, those weights actually get, you know, changed, which is the objective of the training. Okay. Hopefully that makes sense. So, I think this is a good preview of what's to come in terms of the training process and kind of uh, how we um, think about how this stuff actually learns, but you see all of the pieces that we've talked about so far, things like an activation, things like a weighted sum going through the activation. It It's mostly just the components. I don't think you're not supposed to glean anything from these outputs. These outputs aren't very meaningful cuz it's kind of a random input and it, you know, random activation. Uh so, it's not a realistic example, but the point of it is to see kind of the inner workings of this network, to see the different pieces of it. Like typically what's going to be involved in building a network is, you know, setting up uh and activations, doing forward propagation, backward propagation, training it. Like there's training. You kind of see what goes on in the training. It's mostly so we see that inner working prior to us building our own using using the frameworks. Mostly just so we get exposed to that. Uh okay, so let's talk about more Let's talk more about those uh terms we just used, mainly forward and backward propagation uh in our neural nets. Particularly, um you know, just thinking about it from a neuron perspective, data going in um and then uh making those weight adjustments. We'll talk more about that. So, training any neural net really involves two particular phases, and we just saw those in the code really uh involved in that training function, right? And um those two phases have a name, which is known as forward propagation and backward propagation. We're going to break down what each of those really means. Um but it's kind of what the name suggests uh generally is that forward propagation is going to be all about sending data th