Automated Mainframe CI/CD Pipelines with AWS Developer Tools and Micro Focus - AWS Virtual Workshop

Key Takeaways

hello everyone welcome to our workshop before we start the workshop let me introduce myself i'm gilberto biondo i have about 14 years experience with mainframe with different local and international customers working with projects including consolidation and modernization for mainframes i also have some publications around mainframe technology in our workshop today we're going to talk about the automated mainframe cicd pipelines with aws developer tools and microfocus so what should we expect from our workshop today first we're going to talk about why to modernize a mainframe development cycle then we're going to briefly overview the services that will be used in this workshop after that we're going for a demo where we'll see all the services working together in order to deliver an automated cicd pipeline for a mainframe application then we'll move over to the key takeaways and wrap up with a q a session so first uh why to modernize a mainframe development life cycle well the answer for this question may vary from one customer to another depending on their business strategy and where they are in the development life cycle but their answers are usually surrounding three different areas that we're going to cover here so the first area is this loading frame development life cycle and we've seen customers have an increased need and increased space for from business to their applications especially for mainframe applications but their process and their tools have not been able to keep up with this increased pace generating some backlogs also they may have some complex and manual process to provision environment test systems that may slow even further develop the development cycle the second reason is the limited and expensive compute resources we all know that mainframes can be expensive also they are limited by the physical capacity available on site some in some situations the customer also share this hardware between production development and test systems when they do so the resources can be even more scarce and in situations where there is a high demand for cpu for production like the end of the year end of the quarter or other situations the test and development system may receive even less cpu capacity that slows down the processing and the development cycle for your application finally we have the outdated development practices and tools mainframe development practices and future cannot support the modern approach due to monoglyc practices well that can come from a variety of forms that can be the works uh the teams working in silos uh lack of automation and integration lack of team environment team and environment collaboration and other reasons also the archive interfaces may limit the developer productivity even simple tasks such as how to while coding or doing some smart editing may not be available from the mainframe development team so now that we know the three main reasons why customers want to modernize their mainframe development cycle what are the benefits or what can we achieve from performing this modernization we can also talk about three different benefits here the first one is an increase in application development agility so they can develop their applications faster and that can be achieved by simple tasks even as having a modern ide that allows the developer to use out complete or smart debugging or instant code compilation second we have an increasing pipeline infrastructure agility that can be achieved by providing a flexible environment that can accommodate changes in demand for testing and development automatically and dynamically and finally we can increase the pipeline process agility that can be done through the integration of the tools being used and the automation of the process now that we've seen the benefits we can have from a mainframe development like cycle modernization let's see the tools that we're going to use in our demo here we have some different services that are spread of code across some categories including aws services and vendor services first we have the aws developer tools this category will be used to shared source code do the code build test and deploy the applications the first search we have here is the aws code commit this is a highly scalable managed source control that can be used to store your source code data into private kit repositories then we have the aws code build that's a fully managed control continuous integration tool that can be used for building your package compiling your codes run tasks and prepare it for deployment then we have the aws code deploy geda blasco deploy is a fully managed deployment services that automates the software deployment to a variety of instances including amazon ec2 aws fargate aws land and even your on-premise service finally we have aws code pipeline this is a fully managed continuous delivery service it can help you to automate your release pipeline for a fast and reliable application and infrastructure release falling to the next category we have the compute services here we have the amazon let's close compute compute cloud amazon ec2 that provides a flexible compute capacity for our programs and for our systems to run then we have the aws london that allows us to run up our code without even worrying or considering the server instance it allows us to run our codes our script and only pay for the time we consume that resources third we have the amazon ls container registry amazon ecr that provides a simple way to store manage and deploy container images that will be used for the do the building and the testing [Music] moving over to the storage category we use the amazon simple storage services amazon s3 that's a object storage service is highly available scalable with a high security and performance and then we have the additional tools that can be used here to provide an end-to-end solution in our environment here we have the developer id that will allows us to write new code and push this code to the git repository available in aws code commit all the way to the microfox changements emf that will allow us to push the code to the mainframe environment now that we've seen all the searches that are going to be used in this demo let's take a look at the architecture here we have the architecture that's going to be used for this workshop and as i mentioned it's going all the way from the developer ide to a deployed to a mainframe environment our path here starts when the developer id push writes new code and push that to the aws code commit okay so whenever they do that it will be uploaded to the private git repository and that will trigger an aws cloud watch event that will then call aws by black code pipeline remember that the code pipeline is the one that's responsible for orchestrating all the searches and all the steps in your cicd pipeline it's going to first call the aws code build this portion will be responsible for building the code that we have just uploaded code aws code build will run a container with the micro focus build tools this container is currently stored into amazon ecr and will be pulled and provisioned after this machine is provisioned it's going to download the source code out of the aws code commit and perform the build steps those build steps will include compiling and linking the code after the code has been compiled linked and packaged it's going to be stored into an s3 bucket along with other artifacts once this process is completed code pipeline will be notified and will start the next phase next phase here is a call to the aws code deploy that will be responsible for deploying the code for testing the code deployed here will communicate with a microfocus enterprise test server on amazon ec2 using a code deploy agent this code deploy agent will download the arc facts that were stored in the amazon s3 and will leverage all the source code to the correct places for testing when this is complete it will go back to the code pipeline which will trigger the next phase which in our case is the test phase for the test phase we're going to use an aws lambda that will communicate with the enterprise test server to make sure all the tests run if all the tests run successfully it will again notify the code pipeline aws code pipeline that will then trigger the last phase in a project that's releasing to the production servers here we call again another london function that will deliver the code to the target environment okay so this is the architecture that we're going to use in this workshop all the way from the developers coding to the point to a production environment now it's time for starting the demo let's take a look how that works in the aws environment okay so we're now at the aws management console and we're going to take a look at the searches we're going to use for this demo and we will also create us some of the serves that will be required for ease of access i have done some prior work here so the ami rose the lambda row uh lambda functions and the amis have have already been created okay and we are going to work with the code pipeline code commit code build code deploy and calling the lambda functions okay uh before we move on and start creating the new stuff let's uh just take a look first let's go to the ec2 here okay here we have uh one image that's already running it's our test server and we're going to launch a new instance here that will be used as our developer image okay so i'm going to launch image i have one ami ready developer image for workshop i'm going to select this one okay for filter by i'm going to use a z1d instance here 2x large okay instance details uh nothing really special here okay just be careful because since we're going to use this image as our developer image i want to be able to rdp into it so i'm going to how to assign a public ip here okay not going to change anything else just leaving that as default for this storage we have six gig of me of storage that should be more than enough for for this demo you may want to adjust that to your needs we're not setting up any tags either and for security group i have already created one here that has all the ports that we need so we have secure workshop security guru and let's review a lounge everything should be okay here let's launch and i'm not selecting a key pair because i have that already set up as well so launching instance okay let's view the instances okay so while this one is booting up let let me put here developer id okay so oops so this one is going to be the image we will use to write the new code and push the data over to the our ada blast code pipeline okay let's take a look here at the amazon ecr let's container registry okay we do see a repository here ready and within that repository we already have our instance here okay this is the instance we're going to work with it's close to 800 megabytes and we're going to load this instance when we're doing our test and build okay also let's take a look at the london functions and see that we do have the two functions we're going to use in this demo okay uh we have six here but the ones we want to take a look at is code pipeline run fast okay this is the function we're going to run our tests with and this is the one that we're going to uh deploy the code to production okay so we have already the ec2 we have the lung the functions here we check the ecr let's take a look at the s3 bucket as well we have one bit bucket already set up here we have this one mf ed plus cicdr effect workshop okay we're going to use this bucket to store the artifacts we're going to use okay so uh let's start here let's move over to uh a.d blast code commit okay i just click it here we see the repositories and we are going to create a new repository okay there is one already here but i want to create a new one for this demo and for the repository name let's keep that very simple so it will be workshop repo okay a description here this is the um depository for cicd demo okay not going to add any tags and create okay so we have already created the uh our repository called workshop.heap repo uh we have the instructions here to clone this repository uh we don't have we don't have to write those down because we're going to use uh some tool to to clone the repository okay the only thing we want to do here is add a file create a file that will be a dummy file just so we have a main branch here okay so creating the dummy file oops afterwards and putting put my name and email here a commit message dummy file to create a branch and commit change okay so we already have one file here clicking back to repository and back to our workshop dash repo we're going to see our new file here okay very easy very simple so everything we had to do with the code commits already completed let's move now next to code build you can jump from the um left hand side menu here build and click on build projects okay here we also see the projects that we have for this account uh we're going to create a new build project as well we're not going to use the one we currently have okay so for project name keep it simple as well workshop dash okay because this is the build portion build workshop environment okay and uh not building badge not uh enabling to concurrent build okay the source okay so where is the source data that we will use it's going to be aws code commit that's why we created it first okay so uh it's already selected here repository we're going to select the repository what we want it's going to be workshop dash repo and we're going to reference that by branch okay so uh we leave branch here and select main branch okay that means that whenever there is a a change and we're going to look for the main branch into this repository okay we're not doing any additional configuration okay leaving that as these for the environment uh what is the environment that's going to uh build this project we're going to use a custom image here so it's we're going to specify a docker image to use tune environment type uh let's select here we're working with a linux okay so that container image we're seeing is a linux instance so select ins uh linux here okay we're going to pull that image from the amazon ecr okay it's already selected and from my ecr account we're not pulling that from anywhere else if you're doing it if you have multiple account you may want to do that from another account you can just select here okay uh from the which repository we just have one all right so we select here mf aw csvd container repository and that's going to be the latest image we have okay okay we're going to uh let me just here yeah we're going to use code build credentials here and for the service rule we're going to use an existing surface road okay because as i said i have already created all the iam um roles that we're going to need so let's take a look here we already have one okay select it no additional configuration as well for the build spec uh where the build spec file will be located in our uh in our environment okay this is going to be in bank demo um forward slash aws dash ci cd forward slash uh built spec dot ammo okay so this is where our uh build specs going to reside okay if you if it's somewhere else if it's uh the base uh of your environment you don't have to put everything here just put the name of your emo file okay uh for the artifact okay what uh we're going to create a artifacts in the amazon s3 so we select amazon s3 here for the bucket name we're going to use the mfa ws here okay remember that for uh your environment is going to be a different run one right we have uh to create a unique s3 bucket to to use and we're just enabling uh semantic versioning here okay uh no changes here no change here uh no packaging okay so we leave that as none no additional configuration and for logs we do uh we do leave the cloud watch logs here as well okay so now we're going to create the build project uh when we do that it will start a new build we're going to stop it uh okay so yeah no it's good uh so we have the build ready here okay so let's take a look just make sure okay it's it's not running build projects we do have the workshop built here okay uh let's move now next to the code deploy okay for the code deploy we have to first uh create an application okay so we're going to click here in application and create application for name keeping very simple so workshop dash application and compute platform let's take a look uh we are going to use ec2 dash on premise okay so if you wanted to uh deploy that to your on-premise you could do as well or but we're doing ec2 in this case okay now create application okay now we have our application after that we have to create a deployment group as well okay so now we're going to click here on create deployment group okay you can see that it's related to our application here and compute type ec2 dash on premise let's give it a name workshop again dash deployment group okay it's going to use a service rope right to perform the deployment i have already created as well we're going to use code deploy to ec2 okay development type will be uh in place we just have one uh test server that will be used we're not going to use uh being a blue green deployment in this case for several uh for environment configuration we're going to use amazon ec2 instance so we select amazon ec2 instance and we're going to use tags to uh to determine which instance will be running the deploy okay i have that test server it already has a tag okay it's called co-deploy group so we select here code deploy group we select a value value ets cc2 instance okay so we see this one and we can note here that we have one unique match instance click here for details let's just click here to make sure that will point us to our test server so as you can see name is test server so we are uh applying the correct tags okay so uh just to make sure just wanted to make sure that we're uh filtering the correct servers here okay uh you can also see here that you have one uh matching instance okay for the agent configuration okay as edible system manager we're going to uh install now and schedule the update okay it's already installed it's already running but let's leave it as squeeze as is remember that's very important to have the system manager ssm running up and running into your instance in order to be able to perform the deploy okay and you have the code deploy agent as well we're going to use both here okay the the system manager is also going to be used in the lambda 2 to communicate for the deployment settings okay we're doing our ones we just have one instance anyway so we're just doing all that at once here and for a load balancer uh we're not using it okay we're not doing any load balancing now let's create deployment group okay we already have the application and we have the deployment group now let's create a deployment here okay for the deployment group it's already set up right it's the one we just created my application stored in an s3 uh let me just put here uh s3 double slash and ask aws cicd and facts workshop forward slash workshop pipeline forward slash which is that okay and the revision type is a zip it's going to be a zip file and a deployment description this deployment deploys code to test server for testing proposals okay just uh for the additional details here additional behavior we're going to set two things here okay first uh don't fail the deployment to an instance if it fails okay so we uh keep this one okay we select this one and if there is uh already the files with the same name we're going to override the content okay so we override and don't fail the deployment here okay let's create the deployment and now it's going to start a deployment let me stop here because i i don't want to do uh any deployment okay stop it okay we see that it has started and okay so we already have the code commute setup code build setup code deploy setup now let's do uh code pipeline okay so code pipeline let's click on pipelines and create one pipeline as well for the pipeline name keep it simple workshop.pipeline okay it's going to use an existing service role as well we're going to use the one we already have no advanced settings keep it simple let's move to next okay so the first phase is the third phase okay you you see that you have to use the search phase in our case it's going to be a day less code commit and we do the same settings right repository name workshop repo branch name it's going to be the main we're going to use the cloudwatch events okay which is the recommendation here to check for change okay and use code pipeline defaults here for the output format moving next we have the build which is optional okay uh just keep in mind that when you do a pipeline you have the source and you have to have at least one more phase it has to be at least two phases so either viewed or deployed will uh be a must but not both so in this case optional but we're going to use ada plus code com code build here region uh us is north virginia it's the one where we've been using in this demo project name workshop dash builds okay we're going to use a single build here and click next now we also have the deploy you see optional as well deploy provider it's going to be a wsco deploy region us east application name very easy workshop application and deployment group workshop deployment group okay next now we review everything should be ready workshop pipeline for our name effect up artifact location i ended up using another one here it's not a problem it's already set up as well okay but uh just keep in mind that that you may want to update this one first action we're going to use code commit here for the build portion we're going to use code build and here for the deploy we're going to use code deploy i'm going to create this pipeline and i will stop it because i i don't want it to run right away okay so stop execution because i we haven't put any meaningful information to run so far we have source we have built and we have deploy uh let's add two more phases here okay so that's already started uh let's edit and move down here add stage we're going to add a new stage called test okay so we're going to test what we have just uh deployed so add and here we do an add action group okay so this one is going to be test deployed code and the action provider is going to be the lander as we said okay uh north virginia as well uh it doesn't have to have an import artifact here we're going to use the function run tasks and we do have some user parameters here i have already done so place the user parameters that are required for your lambda function to work properly okay uh no variable namespace no output artifact here we're just running some tests done this phase is complete so now it's done as well and let's add a fifth phase here okay or our next phase here which is the deploy to production okay it's going to be called deploy to broad add stage add action group as well same name keep it simple let's use lambda function as well so aws london no input artifacts we're going to use another uh function name here deploy to prod and another set of user parameters okay done as well done here okay so now we have included two more phases let's save our pipeline save and now we see that we have five states okay everything should be ready now we're going to move back here to ec2 let me open a new tab so we don't lose track of the pipeline and we're going we're going to use the developer id here so i'm picking up the ip and going to rdp that into our environment let's connect okay so now we're at the developer ide too okay we're taking a look from the developer perspective uh i do have the enterprise developer up open and running here and i do have one combo file already up and running okay so now what we're going to do is connect our enterprise developer with the um aws code commit and so let's take a look how we we're doing that i have already set up the tools we're going to need okay so we're going to need uh a aws explorer if i remember the name it's already installed here okay uh a day plus two kit i'm sorry so uh we go to preferences and a day plus two key it's you see it's going to ask for your access key id and your secret access key okay so remember i have already created one a um iam user with access to to my account so i'm going to copy and paste here my access key so okay and no worry i'm going to delete this access key and passwords as well okay after this demo so we apply here let's just apply and close it's going to pop up a message here just say yes and let's get back and do the configuration for a day blast code commit as well so we have ada blast code commit it's going to prompt for a user and password let's put our user and password here so user and password apply as well and last thing we want to do is go to region here and select the region we're working with okay so remember that we were working with us east north virginia okay so i selected that apply and apply and close okay okay here okay so we have that configured now let's uh show this so we're going to show view and other and select here a day blue ass um hey they lost explorer i was right so okay now we see a day plus explorer here uh we have code commit let's click on it it's loading it shows our workshop repo right click and clone repository okay we're going to clone it just go next and finish okay it's already cloned uh let's just empty that folder before we can connect our our project here okay so let me just go over there and okay so not now what we're going to do is associate our project with that uh repository so we can be uh committed and push it okay so here at bank demo we're going to right click we're going to team and share project okay so when we click there it's going to uh open up the computer kit repository we're going to click here and select workshop repo which is the one we've just cloned okay finish here okay good so uh we see that now the bank demo has the uh name of the repository here and the branch which domain okay very good and let's open here again the kobold file so uh before we submit our code uh push our code to ada blast code commit there's a couple things i would like to show you here that i find very neat regarding uh having a modern developer id we can use okay so the first thing i would like to show here is that uh when you're working with variables uh you you can just select the variable you want and you can see it highlighted everywhere throughout the code okay so if you want to know okay where where am i using this uh variable you just select it and you see that it's highlighted everywhere okay so that's very easy to find where this variable is being used also uh you know you you might may have a typo or something or forget the variable name so if you put something or you delete something by mistake and this variable does not exist it's going to highlight it for you okay so it shows that uh the red underscore here and if you hold your mouse over it's going to say the operand is not declared okay so it's very easy you don't have to try to build it it's it just pop up automatically for you so so you can fix it okay so i'm i'm clicking here and next thing you can also do is like a control space and it's going to uh show you the options that you have since you i i just missed the g here that was there was only one option and it already auto completed okay so it's very easy it can save you lots of time while coding okay and also it has uh abuse so you see that every time you save it's going to build so you can check it and make sure that it's already working so i saved it it's already compiling my project here and you see that it builded uh it built with no reverse okay so everything should be good we should be ready to uh send that or to commit that to a day blast commit so let's get back here and click right click again bank demo we're going to select team and now we're going to select commit okay uh we're going to see a new uh folder here uh git staging okay so here it's going to show all the unstaged chains uh it's a large number of files because it's the first time we're doing remember that we've just created the repository so it's pretty much empty right so we click the double plus sign here to select everything and stage everything and we're going to use uh use a commit message here this is is the first commit of the project okay not not anything very useful but just to show that in our uh code pipeline okay so it's amazon.com and close it i'm the author and i'm also the committer okay here you have two different options you can commit and you can commit and push okay it's up to you for simplicity i'm going to go all the way i'm going commit and push here already and remember once we do that we're going to upload and we're going to push all the changes or all the staged files over to the ada blast commit code commit and that's going to trigger the cloudy event the cloud watch event the cloud watch events go into trigger uh aws code pipeline and that's going to handle all the uh build deploy and test phases for us okay so just keep in mind i'm going to click here and once we do that we jump right back to the uh aws console to see what's happening okay so click here it's doing and it's done let's close here and get back to our console uh let's move back to code pipeline okay so here we're seeing the code pipeline and it's going to be refreshed and start building the all the phases okay so let's refresh this one we see that the source phase has already completed you see accesses here okay just now it automatically move it over to debut phase okay and while we're doing the build phase you can come over here click in the tails and you can see the details of the your build phase okay so uh let's look here the details and it's currently provisioning the instance remember it's going through the amazon ecr pulling up the image and provisioning it for use okay after it's done it's going to download the source code and perform the build phase okay also you yeah you see that it has already done this download and install face view previewed and everything so it's almost done already you can go to build logs and see everything that's happening so if you want to take a look at what's happening to me during your view you can come up here and see everything line by line and troubleshoot if you have any problems or anything okay you have also the details environment variables and reports if you're doing anything okay we're doing nothing really uh different here so we can move back to pipeline we see that it has already completed and it's on the deploy phase same thing here we also have the details let's click in details see what's going on okay so here we see that it's already in the after install so we have several different phases all the way from the application stop download bundle uh before install install after install application and validate so there are several different steps that you can run different scripts run different commands up to your deploy phase okay you can see everything runs smoothly here so we're good from this code deployment as well let's move back here it's going to go in to update shortly here that the build phase is complete after this build phase is complete we're going to perform these tests okay so now we're going to the tests and the very same thing here okay we can click here in details and watch for the logs see what's happening for the advanced london functions we're seeing from a cloud watch management console perspective okay so we're going to see a few log streams here we have the latest we're going to click here and you see all the things that happen okay so we see that here uh we have a job success so it has already completed all the tasks of all the tests with success and it's going to return here okay so we see the test went fine as well after the test is complete we're going to the code uh deploy to prod okay you see that it has already completed the same thing here you have the details you can just click it and see the same way you did for the uh test phase okay so this is the last phase of our uh pipeline here you can see everything uh completed green as well and keep in mind that you can add different stuff so if you want you can add extra steps maybe you can want to add a manual step or an approval or you can integrate other phases as well okay that's very manageable when you can do that just by going back to your pipeline edit and adding or removing uh stages here okay so you can edit on stage you can add on stage you can add between source and build and deploy and everything okay so that's very manageable on what you can do with this uh move back here let's just see the pipeline and succeeded and you can see that you you can follow a code and also the commit message that was used so you can easily understand what's going on and what's the commit message that was using use it during your phase okay so uh now let's uh just stop our developer id because we did everything we wanted okay so just stun instance here and stop this instance we're not going to use anymore and that's that's the demo okay that's everything we wanted you could see all the way from a developer writing code to deploying the the code to a production environment or to a simulated production environment okay that can conclude the demo let's move back to our presentation okay so now that our demo is complete what are what are the key takeaways from this session okay so uh i believe the main takeaway we have here is the wide range of possibilities that are created when we uh modernize the mainframe applications to take advantage of the capabilities that are available through the aws services and the microfocus tools to create an agile automated ci cd pipeline for the mainframe applications okay so uh based on everything we saw we we now can improve the time to market for your mainframe related projects that can be done through the concurrent tasks and improving parallelism we can also enable an isolated environment for multiple concurrent tests that's also allowed due to the elasticity of the services we're using we're going to uh reduce the test duration while still improving the software quality that can be achieved because we have now alaska resources that can grow and reduce or shrink as we need them and so the teams are not constrained to a set of environment resources to build their tests and finally we have the potential for cost reduction related to test and development this can be achieved by the emulation of the services with the macro microfocus we use it here so uh what are the next steps what we can do next okay so after this workshop we can first experience these services to decide what works best for your needs okay so maybe you want to start very small and just use a modern id or just use aws code commit to store your code and then grow from from there or you want to go all the way through with one single application okay each case depends on your needs then you can explore the additional aws services that can improve your development environment and some samples here we can comment on using um a dayblast cloud formation to provide or to enable you to provide standard environments across different accounts that are easy to set up and destroy as required you can create different aws london functions that will be of any purpose you need maybe you have two different applications that are highly tight and you want to test both applications at the same time so when you perform a change to one application you can also use a lambda function that will also trigger the test and the deployment for the second application and make sure that both applications are working well together finally you can expand the solution to other applications as desired so you can build and then scale to your environment as you need it and just pay for the resources or just grow as required and as you add new applications so that's all i had for this workshop thank you very much now have a session for q a thank you