Python Tutorial : Components of a data platform

Key Takeaways

hi I'm Oliver Wilkins a data engineer and instructed in this field data minded in companies today people are trying to extract value from the tons of data they're gathering they're doing this in an environment called the data platform which is the start of our journey to create robust data pipelines while working through this course you will learn how to ingest data into the data platform using the very modular stinger specification the common data cleaning operations simple transformations using PI spark how and why to test your code and how to get your spark code automatically deployed on a cluster these are the skills you will be able to apply in a wide variety of situations and because of that it's important that you standardize the approach you'll see how we do this note there is a lot to be said about each of these topics too much to fit into one data camp course this course is only an introduction to data engineering pipelines many modern organizations are becoming aware of just how valuable the data they collected is internally the data is becoming more and more democratized it is being made accessible to almost anyone within the company so that new insights can be generated also on the public facing side companies are making more and more data available to people in the form of for example public api's the genesis of the data is with the operational systems such as streaming data collected from various Internet of Things devices or web session data from Google Analytics or some sales platform this data has to be stored somewhere so that it can be processed at later times nowadays the scale of the data and the velocity at which it flows has led to the rise of what we call the data Lake the data Lake comprises several systems and is typically organized in several zones the data that comes from the operational systems for example ends up in what we call the landing zone this zone forms the basis of truth it is always there and the pump oldest version of the data asked was received the process of getting data into the data lake is called ingestion people build various kinds of services on top of the state a lake like predictive algorithms and dashboards for a beetus of marketing teams many of these services apply similar transformations to the data to prevent the duplication of common transformations data from the landing zone gets cleaned and stored in the clean zone we'll see the next chapter what is typically meant by clean data finally per use case some special transformations are applied to disk clean data for example predicting which customers are likely to churn is a common business use case you would apply machine learning algorithm to a data set composed of several clean data sets this domain-specific data is stored in the business layer to move data from one zone to another and transform it along the way people build data pipelines the word comes from the similarity of how liquids and gases flow through pipelines in this case it's just data that flows the pipelines can be triggered by external events like files being stored in a certain location on a time schedule or even manually usually the pipeline's that handle data in large batches are triggered on schedule like overnight we call these pipelines extract transform and load pipelines or ETL pipelines in short there are typically many pipelines existing to keep a good oversight these are triggered by tools that provide many benefits to the operators will be inspecting one such tool a popular Apache airflow in the last chapter good now that you have a high-level overview of the Data Platform let's see how we can you