Develop, Debug, and Deploy Big Data Applications with Amazon EMR Studio - AWS Online Tech Talks

Key Takeaways

hi folks my name is damon cortese and i am a principal developer advocate for emr aws and today i want to talk to you about amazon emr studio so emr studio went ga a few weeks ago it's now generally available and it's a way for you to easily develop debug and deploy big data applications on emr so let's get started if you're not familiar with emr it's a way to deploy open source hadoop frameworks on top of aws so things like spark hive presto hbase these are all frameworks that you can deploy with emr on top of aws and we try to keep up to date with the latest open source frameworks within about 30 days as well so if you've ever deployed a spark cluster if you've ever deployed an hbase cluster emr makes it so much easier and both more performant and cheaper cost of course so what that looks like say you want to deploy a spark cluster you go into the emr console you select spark what version you want to use you select the instance types that you want to use and how many of them and you hit go and we take care of provisioning the infrastructure deploying the software and then keeping that up and running for you as well so that's one of the great things about emr you can do that with all these different frameworks really easy to get deployed there the other thing is there's better performance too at lower cost so with the emr spark runtime for example you can run jobs 2.4 times faster compared to open source and we also recently released some emr runtime optimizations for presto too so just by running on emr you can get uh performance improvements pretty easily as well and of course with faster jobs comes lower cost if you want to optimize cost even more you can also use ec2 spot and reserved instances for your infrastructure in order to get a better cost there as well and then one of the big things about emr and you know the the way that this gets deployed is it allows you to separate your storage and compute so with emr you can read data off of s3 and you can store you know as much data as you want to in there and s3 will take care of you know reliability and redundancy and then you can spin up clusters that scale on demand so what that means is you can spin up a spark cluster with one node you can spin up five clusters with ten nodes each you could spin up ten clusters with a thousand nodes it really just depends on how much capacity you need and you can run those clusters for minutes you can run them for hours or days and so it becomes really really flexible and allows you to scale your compute independently of your storage if you've ever run something like hbase before you know that's really important too with hbase if you ever wanted to increase your hbase capacity you would need to add more spinning disks and more servers in order to host those spinning disks well with emr we have uh hbase support that can read from s3 so you store your data on s3 you start up your hbase clusters and you can size them you know in terms of your actual workload and put as opposed to how much data you want to store on them you can even store start multiple hbase clusters that read data from the same s3 location so you can have multiple readers there's just a lot of flexibility there in terms of scaling your your compute independent of your storage and then speaking of scalability the emr is really easy to scale as well so i mentioned you can start clusters of any size depending on your workload we've also put a lot of work in the past year with emr managed auto scaling so you can start a cluster with this enabled we'll keep an eye on the cluster metrics for you and we'll scale the cluster out and back in depending on the load on the actual cluster so you don't even have to worry about you know figuring out how you want to scale your cluster if you do want to worry about that you can totally do that too you can scale your cluster up in the morning when folks come into work scale it down when they go home well you know um when when when and if we go back home and then um you can even start you know turn them off on the weekends so a lot of flexibility there just in terms of scaling those clusters up and down and emr supports a large number of different instance types as well so you can scale your workload uh vertically if you want you can just you know start it with bigger instance types so that's a quick overview of emr the other thing that's been changing a little bit with emr is just the way that you can deploy it so in the past you might be used to emr on ec2 where you go in there and you pick your instance types you pick how many you want and you spin up that cluster and we again provision all the infrastructure and deploy on all the software and that's been the typical way to deploy emr up till late at the end of last year we also announced emr on eks so if you're using kubernetes at aws you can now run spark jobs using the emr runtime on top of eks this is pretty cool because the model here is more job centric whereas with emr and ec2 it's cluster centric and what that means is with ec2 you would spin up a cluster you would usually have multiple folks using that cluster and you would share some of the resources in that cluster with emr and eks when you submit a job that job is has a specific uh association with the user that's running that job and will get resources from the eks from the kubernetes cluster in order to run that job so that's a pretty cool deployment option as well and then finally emr has support for aws outposts as well so now you can deploy manage and scale emr and your on-premises environments just as you would in the cloud so this is a great option if you have an environment where you know that you have your data center and um you need to run emr in there or you're in the process of a you know on-premises to cloud migration you can spin up emr in there initially and start to do that migration before moving fully to the cloud so there's a lot of deployment options now with emr as well which is really great and then the other thing that's happening is people want simpler environments too people want better development experience and so what we've done is we've launched emr studio as an integrated development environment for um building deploying and managing your big data code so you can essentially build and deploy your code without having to log into the aws console this is a this is something that folks have requested quite often as their corporate environments have gotten larger you can log into emr studio with your corporate credentials they don't need folks don't need an aws console login you can start notebooks jupyter notebooks in seconds and run that run that jobs later and you can also build production pipelines uh with emr studio as well and i'll show you how to do that a little bit later on and then finally one of the big benefits here is you can save debugging time with native application uis inside of emr studio and i'll talk a little bit more about that too in terms of how we've made that easier so that's emr studio we're trying to make it easier for you to get in there do your job and get back out and without any further ado what i want to do is i want to hop right into a demo of how to create a studio in emr so let me um let me hop out of the presentation i'll swip swap swap over to my browser really quickly all right so here i am in the emr console if you've used dmr before this is pretty familiar to you i've got a list of all my clusters right here i've got one running and previously if you wanted to start emr you would go here and click create cluster and this interface is pretty straightforward right you name your cluster you select what release you want the instance types you want and then your ssh keys now if you want to switch over to the advanced options that's where it gets a little bit more complicated you select all the different types of software you want and then you click next and you get into the section about instance groups and instance fleets and all the different mix and match of instance types so it can get pretty complex pretty quickly right and some folks are used to this and they're pretty good about you know knowing how to configure this and in most cases you're probably only going to go through this process a few times but like i said we wanted to make that easier so let me get out of this interface here and let me go to emerge studio you'll notice a new emr studio link on your emr console and when you click that that takes you over to the emr studio console now what we do on that getting started page i mentioned we do have aws single sign-on integrated with studio so that means that you can import your corporate credentials into sso we support active directory and other third-party providers as well so you can import your corporate credentials there so we just do a quick check in emr studio to make sure your sso is enabled we also integrate with service catalog so what you can do is you can actually create cluster templates that folks can spin up really easily inside of emr studio and then you just go ahead and create your studio so if i click on that this is a one-time setup to go ahead and create your studio so you might have a data science studio or a big data studio or something like that you can create a description there when you first set up the studio you need to define a few things that get used throughout the creation of the rest of the workspaces in the studio so there's a service role this takes care of creating you know emr clusters and things like that and then this is a user role so these are roles that folks these are permissions that folks get when they sign into the studio and you can add additional permissions too to kind of scope that down a little bit and then you choose an s3 bucket so when you're creating a studio this is where all your studio notebooks live those all get saved up to s3 so you can open those and close those and have them saved and then you just select your vpc and subnet this is if you're spinning up clusters or accessing emr clusters the studio just needs to be in the same subnet as your emr cluster so you go ahead and do that and then you click create studio i've already gone ahead and created a studio so let me go over and show that studio so when you create that studio the next thing you need to do is add users to that studio so if you go into view details what you can do these are all the details of the studio and then you can add users down here and this pulls in from aws sso again so i've got my regular daemon and my basic daemon and i can go ahead and add basic daemon and then what you can do in there i get added as an administrator by default but you can change the policies that are assigned to each users so i have a basic and intermediate policies there's my emr studio basic user policy and then i also have an intermediate policy and we'll talk about what the difference is between those in a little bit so you can assign different users there so that's creating the studio and granting users access now let's hop into the studio to see what that looks like so i'm going to go ahead and here and you get a studio access url now if you're already signed into aws sso you'll be logged directly into the studio if not you just sign on with your corporate credentials and you get um put back into your studio dashboard here there's a couple things you can do here you can create a workspace or you can also manage your existing clusters so let's uh look at our clusters real quickly so i can see my emr and ec2 clusters and this pulls back the different clusters that i have running on emr and ec2 you can see this is similar to the emr console where you get your cluster id the name and different information about the cluster the other really neat thing is i can click on a cluster here and depending on the applications that are installed in that cluster i can launch my different application uis straight from emr studio so the reason this is awesome is previously if you are using emr and ec2 what you would typically have to do is you would create your cluster and then you would ssh into the cluster to be able to access those those cluster uis or you'd have to set up the appropriate networking between your corporate environment and the vpc that that cluster was running in in order to be able to access those cluster uis well with emr studio now you can just select your cluster here and you can launch the application ui and i just launched the spark history server so this is the spark history server for that specific cluster that i was just logging into and now i can see i ran an application on that cluster so i can click through to that application and i can see you know information that is about that application so i can see what that application was i can dig into the environment and the executor so i can kind of debug that that application and see what happened and so that's much much easier than ssh into a cluster and port forwarding so that's one thing that we've made a little bit easier in emr studio is just the ability to launch those application uis and again that's an off cluster ui you don't have to sshn or anything like that and you're authenticated through your corporate credentials if you're using emr and eks you can also manage your emr and eks clusters there so i've got one emr and eks cluster running here i can click through there and i can also see all the jobs that were running on that cluster one of the other cool things about emr and eks this is a job that completed and i can actually go here and i can launch that spark history server and that'll pop that open over here and this is a completed job that's already done but i can go in and i can debug that job after it's already oops let me go back and click that again so i can launch that up and i can go in there and i can debug that job after it's already been completed so that's one of the nice things too about this is both on emr and ec2 and emr and eks i can launch those application uis without having to you know kind of connect in an ssh or whatever and it's really easy you can see i'm connected to that application ui now i can go in there and i can debug this spark job just like i was debugging on emr and ec2 so pretty easy to go in there and launch those different application uis directly from emr studio so that's the one big thing about studio is it makes it so much easier to kind of look at your clusters and look at your jobs and be able to see some really nice details about those clusters and jobs but let's go back to the dashboard and what you probably want to do in emr studio is create a workspace so in order to do that this will be a fully managed jupyter notebook and you can create your workspace here you can just say scratch or whatever and then you select the subnet again this is where the emr cluster will get created now the other nice thing about emr studio is it makes it really easy for you to connect or create clusters so down here under the advanced configuration one thing you can do is you can attach to existing clusters so whether that's emr and eks or emr and ec2 you can go there and just select the cluster that you want to connect to so that's pretty easy if you have um intermediate or advanced permissions for emr studio you can also use respectively cluster templates or create an emr cluster from scratch cluster templates are really nice these are cloud formation templates that are imported into service catalog and so for example i've created this matplotlib cluster that has matplotlib pre-installed and a bunch of other different software pre-installed and i can go ahead and select that cluster template and if i want to i can have a set of parameters that the user can choose for that cluster so i can change the name or the emr release or i can just have no parameters and say go ahead and start this cluster so cluster templates are really useful if you've had somebody that's gone ahead and done a lot of the hard work in terms of configuring emr and creating that cluster template and then other people can reuse that so that's really nice if you're familiar with emr or you just want to be able to create an emr cluster from scratch you can do that too again this is much easier you just go in here you enter your cluster name the emr release we take care of provisioning the right applications for you and you just say how big you want that cluster to be and when you create the workspace we'll go ahead and create that cluster for you and attach that notebook to the workspace so i'm just going to create a workspace and attach it to an existing cluster and i've already done that and you can see my scratch workspace here so let me go ahead and launch that workspace now what this does is this is going to open my jupiter lab notebook for me and so what happens here is this pops me into jupiter lab and connects me to my emr cluster and that might be a cluster that's already started or it might be a cluster that um that i just created and let's see if we're connected so over here on the sidebar you can see that i'm already attached to my ds matplotlib cluster if i want i can detach from that cluster and attach to other clusters so that's pretty nice that makes it really easy but when i first land in my studio here i've got my little scratch notebook and a launcher so i can launch different notebooks with python 3 or spark different spark kernels i can hop into a console the other thing that we have in there are notebook examples so if you're not quite sure where to get started just click on that notebook examples command and we've got a bunch of different notebook examples stored in github that will show you how to do different things so pandas or matplotlib or querying you know data from s3 depending on what you want to do we've got a bunch of different examples in there when you first launch though you get this this default notebook that's already connected to my cluster and already running python3 so i can just go in here hit print ok and i can do all my data science stuff or data analyst stuff in here and of course this is just a standard jupiter notebook so i can do my my markdown i can do my python code i can switch to a pi spark kernel if i wanted to so that is how you can get started really easily with creating a studio creating a workspace in the studio and then creating your your own workbook in there so pretty simply get started there now one of the things that definitely is common is folks want to be able to connect and collaborate with other people that they're working with and one common way of doing that is simply in github so i've got a repository out here of demo code and so this is my demo code repository and in there i've got an emr studio notebook that i want to be able to pull in so i'm going to go ahead and just copy that github url and go back to my emr studio so in studio what you could do is you can actually add a new git repository if i go in there i can do damon's demo code i can add the repository url and the branch name and then i can connect to this repository without credentials or i can even create a new secret or use an existing secret and this in this case this would be a personal access token from github so i could connect as my user but i'm just going to use a public repository without credentials i'm just going to go ahead and clone this so when i hit add repository that goes and adds that new repository and i can go ahead and link that repository with this with this workspace so i've done this before you can see my old demo code repository there so i'm going to go ahead and link that one and what what's happening there is in the background we're doing a git clone we're pulling that repository into s3 and then you'll have the information in that repository available in here so if you wanted to pull your repositories into here you can go ahead and do that we support github bitbucket gitlab and if you have a privately hosted git repository as well we also support that so that takes just a second to clone depending on how big the repository is and then if i go back to the file browser here what you'll see is now there's a demo code folder so i'll hop in there i'll go on demo code emr and then studio and there's my notebook that i want to pull in so let's go ahead and open that notebook and this is just opening it straight from the github repository and so this is a notebook that i've built and what i'm going to do is i'm going to build a weather map of the weather for a given day in the u.s so i'm going to install a bunch of libraries in emr studio you can do a pip install if you want so you just go ahead hit pip install those requirements are already on this um uh this notebook so that just went ahead and did that super easily and what i'm going to do is i'm going to use the registry of open data so if you haven't checked this out before this is pretty cool with aws there's a registry of open data and there's i think over 100 different data sets of freely available data for you to use and one of those is this era 5 zar data so this is um you know different types of weather data like precipitation or pressure or temperature and what i'm going to do i'm going to take a few of those data points and build a map out of those so this just shows me reading data from s3 there's this era 5 pds bucket and that just shows me opening that bucket and what i'm going to do is i'm going to build this weather map for a given day now the main thing that i want to call out here the reason that i'm doing this is i have this one cell here that has a weather date in it so i'm going to say okay fine let's draw a weather map for january 13th 2020 but the thing about the cell is it has a parameters tag so if i click the wrench icon over here and go down a little bit oops let me click on that cell you can see the cell has a parameters tag now what this means is that i'll be able to override this parameter as part of a data pipeline i'll show exactly how that looks a little bit later on but this is how we can actually take notebooks and run them as part of data pipelines which sometimes blows my mind but is something that is that can be pretty useful so i'm going to run through this notebook really quickly i set the weather date these are just a few helper functions to read in my data from s3 this cell right here this opens the czars and loads those into different variables this just formats them and then down here this is my matplotlib code that is actually oops going to draw my weather map so if i execute this cell the old weather map disappears and there's the weather for january 13th 2020 you can see it was pretty cold looks like there's a cold front coming down into seattle or something like that and so that's how you can easily pull in um notebooks from different github repos now if i wanted to i could save this and i could actually go back to the git section here and you can see the repository that you're on and the branch that you're on you can even create a new branch if you want to so maybe i'll create this and say feature bayman's demo we can create a branch directly in emr studio and i change this notebook maybe i uh let's see maybe i change this notebook a little bit and i say let's do it for 2021 how about april 1st what was happening on april 1st so i'm going to go ahead and run that cell and all below and that goes through and pulls in all the data again and in just a second we should have our weather map for february or april 1st of this year you can see it's a little bit warmer and so let's say i want to commit those changes so i can just go ahead take that changed file stage it and then add my commit message commit for the win or for the queue and go ahead and commit that and so that commits that changes that's not pushed up to the github repository yet but what i could do is there's a push button up there and i connected to this repository without credentials so it's actually going to ask me for credentials before i push i'm not going to push right now but this is how you can push back up to the repository so if you're working with folks you can make these changes you can push those changes back up and that's how you can work with github inside of emr studio as well so that's something that makes it pretty easy to collaborate with your peers and then the other thing that i want to do i mentioned we can also parametrize these notebooks there's a couple things we need if you want to take this notebook and run it as part of a pipeline so the first thing that i'm going to do what i need to do is i need to go back out to my emr studio and i need the cluster or the um the workspace id and the cluster id so i'm going to go ahead and pull out that information and so in here here's my workspace id and there's my cluster id so in order to run this notebook through a pipeline i need those two bits of information and i've got a little bit of sample code over here to go ahead and run this so i need my editor id and cluster id and i've already got that and what i'm going to do is i'm going to use the aws emr start notebook execution command and i'll provide the editor id like i mentioned here's the cluster id down here and then i can provide a parameter to that notebook so i'm going to take that weather date parameter and i'll replace that with this 2019 0901 parameter and then you specify the path to that notebook and so what i'll do is i'll say there's my demo code emr studio weather day notebook so i'm going to take this and i'll go ahead and execute this in my in my console here i'll go ahead i'll hit enter there and that will submit a job to the emr api and give me back this notebook execution id and then what i can do is i can take this notebook execution id and describe notebook execution and what we should see is we should see this job starting up on the emr cluster and so this is a way that you could integrate your notebooks with a data pipeline so for example you can take this notebook you can go ahead and submit it and you could do this with the command line you could do this using the api if you wanted to if you had airflow up and running you could actually make this part of an airflow pipeline too where you could start notebook execution and provide these different parameters and then wait on that execution to finish and then once it's finished you could you know do other things with that pipeline or just mark it as completed so this is running right now we can see that that status is running and when this is completed what happens is there's an output notebook uri so the emr takes this notebook runs it through the cluster and then writes out another copy of that notebook out to s3 out to your emr studio storage bucket if i do one describe notebook execution you can see that that's finishing up and what i'm going to do i'll actually do an s3 copy of that notebook i'll copy that locally and then what i'm going to do is i'm going to upload that back into my studio so we just downloaded that let me go back to studio and i'll open this up i'll go back to my folder back down to the root here and what i'm going to do is i'm going to upload that notebook that i just did and so this ran the notebook through with that updated parameter so if i go ahead and open up this notebook and scroll down a little bit what we're going to see down here is there's my original weather date cell and then here is the parameter cell so this is the parameter that i passed in through the api so emr inserted that into the notebook and then it went through and executed the whole notebook and if we look down at the bottom now we have our weather for september 1st 2019. you can see there was actually a big hurricane i think that was hurricane dorian off of the coast of florida on that day and so this is how you can run these parametrized notebooks too as part of a pipeline and this one it just you know dumped it out to the um the notebook itself which may or may not be useful maybe you are creating notebooks for other folks to consume but you could also use this to write out to an s3 bucket or you could call out to an api with different data right back to a database so there's a lot of different things that you could actually do with this notebook so that's a pretty nice example of taking that code running it through parametrizing it and then building it into your pipeline and so i've got my git repo here i could add that back into the git repo if i wanted to but i don't necessarily need to to do that right now but again i think it's you know so easy to get started here in emr studio if i go back to the launcher like even just taking one of these sample notebooks i can pop open this sample notebook right here and you get a read-only copy by default but then you can save it to your workspace and this just creates a copy in your own workspace that you know you could then commit and let's see if i go back and here and i can just actually go ahead and run this whole notebook for me so i'm just going to say run um let's say run all cells and so this goes through and starts just running through all the different cells on this notebook and you can see it's popping open you know another matplotlib chart here and it's using pandas to kind of pull things down so there's a set of libraries that are installed by default on on emr studio but if you want to you can install your own um you know pip libraries you can just do a pip install whatever you can install your own different information and so pretty easy to get started there save this back to your workspace commit it up to github so it makes it pretty easy to get started in there the other thing i do want to call out we have some demo examples so i've got my demo code repository here on github that i showed we do also have in the aws samples repository we have an emr studio samples repository in there and so in there there's a couple different cloud formation templates so this can help provision an emr studio for you it gives you the iem roles that you need and some of the security groups that you need to get started with emr studio and then there's also some cluster templates in here so these are sample cluster templates you can import into service catalog in order to see um you know how to how to get those clusters up and running again these templates are pretty small this is a 50 line you know service catalog or cloud formation template but it makes it really easy if you're back here in emr studio again in the cluster section not only can you attach to an existing cluster but you can also create clusters from within the workspace as well so you've got a really simple interface there for creating a cluster or if you go to the cluster template again you can create cluster template from within the workspace too so you can spin that up use the cluster template with a certain set of parameters and then detach in here and connect to it the other workflow that does come up pretty often is people may have development clusters that they work on and then they may want to run their job on a production cluster that's either larger or has different sets of data on it so you can easily detach and reattach to other clusters there too so pretty cool to get up and running um a couple other things i just want to call out so there's the emr studios samples if we go back to the jupiter lab and back over here in the launcher i mentioned the notebook examples those notebook examples are also on github so um these are just the same examples that you'll see in in emr studio we hope to add more examples here as we hear from folks so definitely as you um you know work with emr studio if you feel there that there are examples that would be useful uh definitely feel free to reach out and let us know and then you know once you're done with that repo you can disconnect that once you're done with the cluster you can shut it down and then that is you know how you can get started with emr studio so again you hop in there you've got this dashboard you can easily create a workspace with your clusters you can easily manage your emr on ec2 clusters and all the different application uis on there as well you can easily connect to github and push changes up to github or pull changes down and you know collaborate with folks from that way and that is how to get started with emr studio so something we're really excited about um you know the goal here is to make it easier for folks to be able to get into their work um you know do that analysis make it easier to run emr clusters make it easier to run emr on eks and also debug those jobs a lot easier too so um that's that's kind of it for today um you know we're gonna take some some q a for a little bit so if you have any questions uh definitely let us know the other thing i do want to mention feel free to reach out to me at any time i'm at the court on twitter and uh dot dev is my website so go check that out i've got a youtube channel where i'm putting out more videos about emr studio and just emr in general and analytics in general so feel free to subscribe to that and i just want to thank you all for tuning in today and let's um let's see if we got any questions all right thanks everybody