Microsoft Rocket: Hybrid Edge + Cloud Video Analytics Platform

Key Takeaways

hello everyone and welcome to today's webinar before we begin we wanted to cover a few housekeeping items for any common technical issues please refer to the help widget located at the bottom of your screen if you have any questions during the webcast you can submit them through the q a widget questions will be answered immediately following the presentation any resources and links provided from today's presentation can be found in the resource list an on-demand version of the webcast will be available approximately one day after the webcast and can be accessed using the same audience link that was sent to you earlier hello everybody welcome i'm ganesh ananti narenin i'm a principal researcher at microsoft research over my career i worked on a broad set of areas in systems and networking ranging from mobile computing to big data systems to wide area networking and offline video analytics which is the focus of this webinar so today i'm going to talk to you about our video analytics platform that we call rocket that's a plugable extensible platform that runs across the edge and the cloud and why are we doing video analytics frankly because there are cameras everywhere all the way from government enterprises being deployed for surveillance for traffic optimizations to private enterprises in private buildings in retail stores all over the place a startling piece of statistic is that there is a camera deployed for every eight people in the us that's the scale that's the the scale at which we're talking about here and what do people do with these cameras today a variety of applications are actually powered by these cameras the first set of applications that you know some of this that we ourselves have worked on one is on deployed by restaurants where they want to be informed when a car is pulling in or when people are walking in so that they can either dispatch ushers also like pre-make certain food items retail stores are another class of applications which is also where video video cameras play a big role people want to know both from a historical perspective what's like the heat map of people moving around so that they can help with product placements as well as sort of like real time intervention when they see certain customers that need help or assistance smart cities and urban mobility is another vertical we've actually ourselves done a lot of work in smart cities and urban mobility where smart you know video cameras are used for both traffic congestion as well as traffic safety let me tell you a little bit more about traffic video analytics which as i said has been a driving application for our project on video analytics we call the work video analytics towards vision zero and for those that are not familiar vision zero is an initiative that started a couple of decades back in sweden with the objective to eliminate pedestrian and biker deaths that occur due to traffic fatalities and during the time we were starting with the project we reached out to the nearby city of bellevue in washington where they had just signed on to the vision zero objective as well and and they were interested in using their already widely deployed traffic cameras for traffic safety and traffic efficiency and in fact the startling statistic that caught our eye was the fact that traffic related deaths are among the top 10 causes for deaths worldwide and so we were we've entered it that hey if we can make a dent at this if we can make a difference to this that would be great and and so we did a bunch of work uh you know especially in terms of using the rocket video analytics stack towards analyzing the bellevue traffic cameras where we get a bunch of things for traffic volumes for traffic planning and so forth and let me show you a short video of the kind of stuff that that video analytics can power in the space of traffic video analytics microsoft research is working on a multi-national road traffic safety project called the vision zero initiative started in sweden in 1997 vision zero aims to eliminate fatalities and serious injuries on the road here we show how a microsoft edge computing device can save lives by analyzing a camera feed in real time when a person in a wheelchair is detected on the crosswalk the device resets the pedestrian lights timer allowing the person in the wheelchair to safely cross the street as part of this project we partnered with many cities all across both america and canada and this helped us understand both the real use cases that they had as well as the challenges towards achieving this solution are we quite glad about the fact that this project won two major national awards towards the fact that it was uh it was towards the fact that it was a visionary project uh in showing what can be done in terms of traffic safety and traffic efficiency with video analytics so to summarize our goal is to democratize video analytics and what that means is that we want to do video analytics that is on live videos so it functions in real time it is low cost in that the cost of running it is not prohibitive especially if you have to run it 24 7 while at the same time the outputs are still accurate a bit of context on the fact that the real reason why we got interested with video analytics was the fact that there was so much excitement around what the vision community had achieved in terms of how accurate the results they could produce for object detection object classification and so forth and and we just wanted to bring that out to the general public and and at that point we had seen that the cost of doing it was really high and it was not something that was suited for running in real time and so that in a nutshell is our objective towards democratizing video analytics this is the rocket video analytics stack right on top as you can see in orange are the fact that there are various application verticals some of it which i just explained in the early part of the webinar ranging from smart cities to connected restaurants retails and so forth and and and the the the framework allows for expressing the needs of this application in a very generic fashion so you can construct a bunch of modules put together as a pipeline that then gets passed down to a pipeline optimizer the pipeline optimizer does its job of picking like the best choices and applications that is needed this is act into a query optimizer if you would in a database that is then passed down the layers where resources are allocated to it and then this pipeline is run both across the edge and the cloud and finally what we do is we take the outputs of these live video analytics and pipe them to a database so that if you want to do any kind of historical analysis later on that is something that you can go and do without spending too much time and it can be interactive the cool part is is uh is that you know while i would finish the first part of the webinar the second part we actually are open sourcing this code and my colleague would be giving you a walkthrough of how to use the code and the various components in the rocket video analytics stack so so to just give a larger perspective of what the video analytics stack exposes you know you do see the software stack right in the middle there but then on top we make it easy for you to have very common functionalities that all video analytics applications want which is a way by which we can do easy filtering and getting all the moving objects in a frame how do you decode all the frames in a video how do you track the objects across the different frames how do you plug in various dnn models uh from your favorite dna execution framework like tensorflow and how do you put all of these together so that you can construct your application needs and of course this is the basic building block and the blocks that you see on green on top takes these outputs from these building blocks in the rocket video analytics platform and then builds up higher level applications like counting traffic directional counts across time how do you assess near collisions that happen how do you get vehicle speeds and so forth so today's webinar would cover two major parts of the rocket video analytics stack and i'll do a deep dive off the technical components the first thing is how do we use approximation as a key tool for us in being able to run scalable video analytics across the edge and the cloud and the second part would be switching gears a little bit and talking about interactive querying on stored video feeds let's get to the first part here as i explained earlier the rocket video analytics platform allows for users to exp to express their interests their applications as a series of pipelines put together this is an example of a pipeline where there is a decode where there's an incoming stream that's decoded and then there is a set of frames that are available these frames could get resized you might want to do frame sampling and not process all of them then you want to do some kind of background subtraction so that it does motion detection for you to know if there are moving objects at all it's a very effective way to filter out certain frames if two frames are repetitive and have the same content and then you go ahead and pass it to various you know dnn models like like yolo and fast rcnn and so forth and these configurations are pretty important like the various choices that you make here what's the resolution what's the frame rate that you go to what's the object detective model that you're going to use they have a huge implication in terms of of of the amount of resources that is consumed what's the accuracy of the output as well this is an example of what happens with say the frame rate we see that with increase in in the amount of frames that you don't process and sample out of course the cpu cost reduces but then at the same time the quality also drops and we see a similar trend for resolution as well more the resolution that you process at the better is your output but at the same time the costlier is your processing uh this is an example of say two you know of one of the videos in in the city of bellevue where we see what the output looks like for an object detector for two different versions of yolo clearly we see that tiny yellow which is a much cheaper version of the original full-blown yolo does pretty well in this intersection which is which is fairly sparse not very cluttered however in the same city you go to another intersection what we do see is the tiny yellow no longer looks very good it misses a lot of objects it finds objects that are on the sidewalks to be cars and it's quite confused here whereas a full-blown yellow does much better and so putting it all together you know these various knobs that we have the various implementation choices and the dnns that you have how much do they make an impact let's look at this graph here this graph shows on the x-axis the the resource demand that you need for running these these various pipelines note that this is a logarithmic scale graph and of course on the y-axis is the quality or the f1 score and what we see here is that look at these two points almost the same quality of the output but with two orders of magnitude difference in the amount of resources that they need and this general trend is true for variety of applications including a general object tracker a dnn classifier that you might want to use and and why this difference it's just simply because it's it's dependent a lot in terms of of you know the kind of camera that you have the lighting conditions at that moment the color of the objects and unlike sql queries unlike database queries there is no analytical model that is available for us to construct these resource accuracy profiles and not only do we have to get these resource accuracy profiles we also then have to execute these pipelines across this hierarchy of compute options that is available there's computer available on the camera there's compute available in edge clusters which is deployed by these enterprises and then of course there is the public cloud as well and there is the network connectivity across them in fact i would encourage you to check out this this fun article that we wrote a couple of years back where we were predicting that real-time video analytics is actually would be the killer app for edge computing so to summarize the problem statement here we want to pick the query plan to go with the database analogy that says what are the various knobs choices that we have to make what are the implementation choices what are the dnns that we use and then place these different modules across the hierarchy of clusters and we have to do them jointly so this is the problem statement more specifically decide the query plan and placement and what we want to do is when there are multiple such video pipelines that are running we want to maximize the aggregate quality say the average quality for instance and then run them across this hierarchical setup and of course there are diverse quality requirements for these different applications like for instance the the tolling application might not really have strict latency considerations but would need very high accuracy because it would not be acceptable if you get told inaccurately traffic lights for instance is an example of something where you need moderate to high accuracy but you do need to happen it within a second or two amber alert could be an extreme example it needs not only high accuracy it also needs to be super real-time so the framework allows for setting the various requirements that these applications have to provide an overview of our solution what we do is that when a query or a pipeline comes in what we do first is run it through a profiler what the profiler gives you is this resource quality profile note that we've seen this resource quality profile gives us the opportunity to make very good choices in terms of how efficient we can be without dropping accuracy too much and then of course as i said different applications have different requirements for what the quality is that they want so these queries uh you know all are all passed to a scheduler the scheduler of course is the brain that goes ahead and and decides which uh decides which implementation choices to make for each pipeline and how to run them across the hierarchy of clusters this leaves us with two main challenges the first one is how do you efficiently generate this resource quality profiling note that i had said unlike sql queries we don't have analytical models that are available for for video pipelines the second is how do you schedule for for aggregate values of quality and still do placement at the same time so the first part is of course done offline and the second part is the one that is done online on a continuous basis so let's first move to the resource quality profiling this is an offline operation where the objective is the following for every you know query plan so to speak which is every combination of configurations that we have you know say for this specific resolution this specific frame rate this specific dnn choice what's the resource consumption going to look like what's the quality of the output that's the profile that we need that the 2d graph that i basically showed you of course a trick here is a key problem here is that the the quality is something that you can measure only with a ground truth and for a general query in the wild that just comes in there are no hand labeled data sets available so one thing that we do is we use data results from what we call a golden configuration so this is like the most expensive configuration that you can choose we use that as the gold standard and then measure the results of all the other configurations against this golden standard so what we primarily do during the profiling process is to have a targeted search for the most promising of query plans where we don't where we don't necessarily profile all the options on the entire video set that we have but we try a small set we try to see if these configurations are promising if they aren't we let it go and if they are promising we try it on more videos to see how that goes uh but the but one one challenge in this is this profiling is not something that you can just do upfront once and then you're done what we really need is periodic profiling because the video content keeps changing there are times for instance say in a traffic video where there are a lot of cars there are other times when there are not that many cars if you recall the very example that i gave earlier when things are not very cluttered something very lightweight works just fine and we want to identify and exploit those opportunities this is an example of say what would happen due to the value of say periodic profiling in this graph here on the y-axis is is what is the resource consumption that we need and the x-axis is is is roughly the accuracy of the outputs that are produced clearly we want it to be more in the bottom and more to the right and and the the set of blue and the red dots clearly will show you that doing just a one-time profiling uh ends up with worser options than doing something much more periodic so that's great this shows the value of periodic profiling however i do a little bit of a cheating there in that in this graph i have actually not included the cost of profiling when i do include that look at what the results happen we end up with much worser uh you know values in the fact that the cost due to smarter choices is all wiped away for the fact that we've paid additional cost in doing this profiling to get these choices at all so in a nutshell now the key challenge in this is to reduce the profiling cost i wouldn't go too much into detail but we use three key techniques here the first one is temporal correlation where if a certain configuration is bad then it continues to be bad for a long time likewise if a configuration is good it does stay good for a little bit and so we don't have to keep profiling them all the time so this helps us exclude a vast number of configurations out of the picture the second one is that organizations rarely deploy cameras standalone they deploy a fleet of cameras say in the city in the building in the store and so forth and there is a fair bit of spatial similarity in the profiles of these configurations what we've seen is is we can cluster these cameras and do the profiling for just one of the cameras and pass this profile to the other cameras and that holds just fine this dramatically reduces the cost and the third is the independence of these configurations where we've seen empirically that these configurations tend to operate independently so we don't necessarily have to do the combinatorial thing of profiling all the possible combinations of configurations but can do profile these configurations individually i encourage you to check out our sitcom 2018 paper for more details on this profile so essentially the profiling uh goes ahead and as i said generates this 2d plot of the quality versus the resource demand you know very smart about the fact that the profiling has to uh you know not consume too much resources but then at the end of it now we end up with this with this 2d graph which then has to be passed to the scheduler and often there is a lot of options that are now available so now what we do here is that to make the scheduler's job easy what we do is is select you know the pareto boundary of these these options like for instance take these two points the green point and the red point note here that the red point is strictly better than the green point in both dimensions there's no trade-off here in both quality and resource demand the red point is better than the green point which means that we don't have to consider the green point at all as long as we have the red point this is a point that we call pareto optimal it's a concept from economics and if you generate a boundary of all these pareto optimal points you get the pareto boundary and so the non-pareto plans are no longer required for us because as long as we have some point uh on the pareto boundary we're good enough so that's the thing that we then pass to the scheduler in the next stage and this note that this dramatically reduces the amount of options that the scheduler now has to deal with so this brings us to the second part of the talk in terms of the scheduling itself and and note that this is a multi-resource scheduling problem i've been sort of like presenting these graphs for uh for simplicity as just cpu resource or just a compute resource whereas whereas this is a problem in a hierarchy of clusters that includes not just the compute but also the networking connection between the cameras to the edge to the cloud and so what we do is we use the the idea of a dominant resource demand for each combination of configurations and placement and this is a concept that actually was uh was was roughly now generated during the the time when big data analytics was happening for multi-resource scheduling and so so what this allows is that uh you know the dominant resource demand it avoids lopsided drain of any single resource at any location so the dominant resource demand uh you know make sure that that we are using all the resources at all these these various locations in an even-handed fashion and so one week once we get this this dominant resources then we essentially use a pareto band that ensures that we don't have to go through this entire search space but only have a small fraction of the search space in terms of evaluation we've evaluated our system with a workload that consists of traffic cameras and surveillance cameras these original frame rates varied in in terms of uh you know how much high risk they were what the frame rate etc was and we used a variety of queries which we again you know we've seen were important for for lots of video analytics applications and just to summarize what we see was that we can get considerably better accuracy than competing schemes and despite the various approximations that we made to the scheduling heuristic we are still pretty close to what an optimal would do so this finishes the first part of the talk where i showed you you know video analytics can be done across the edge in the cloud and how approximation is a key tool that we can use now let's switch to to using these the fact that we can do efficient live video analytics to say how can this power interactive video analytics on stored videos on historical videos after the fact and why do we want to do it simply because video recordings are ubiquitous it's done everywhere and you know being done in terms of highway systems at homes even on our phones and and today you know it's it's great that we have these cnns these convolutional neural nets they do enable accurate query for instance you know traffic planners in the city of bellevue might want to know find all the trucks in the video yesterday in the last week because this helps them know how the streets are being used and so forth and if you want to do it it's great you know this is kind of what the outputs would look like it gives you all the trucks but you want to do this today it is slow it takes a lot of time and it is quite expensive and sure so why so let's fix these two problems you know we don't want it to be slow because then it's no longer interactive and there is enough to show that interactivity is quite important and we don't want it to be costly either so what can we do about this well one way to make the queries fast and not not them be slow is we can analyze all these videos at in just time just on the live videos let's analyze them and plug them into a database but this is a very expensive solution partly because enough studies have shown that a lot of the videos never get queried so this end up being a bunch of wasteful expenditure expenditure that's close to 400 a month per video stream of course at the same time you can address this problem by just doing nothing at ingest time but analyze all the videos at query time but this also takes a lot of time for like a month long video it can take many hours and month long queries are quite common for planners it can take many hours and this is hardly interactive so so this is kind of what we want to provide which is enable low latency query low cost querying and at the same time not lose the fact that we still need high accuracy results and we wanted to lose our large historical video data sets graphically speaking this is kind of what we want to look at if you look at these two axes the ingest cost and the query latency what we want to be doing here is that of course we want to be doing better on both of it which is closer to to to the you know to the left hand and the bottom for both these dimensions and even ingest heavy scheme where we do all the work at interest time falls here which case it's a great you know kind of query latency and if a query heavy thing where we do all the work only at query time when the user asks for it falls here we ideally want to be here this is kind of where our goal is and towards achieving this goal there are three main objectives which is let's provide a low cost indexing let's still make sure that we achieve high accuracy and low latency at query time and finally as you can guess this is a system that that needs us to make these hard choices these trade-offs between ingest cost and query latency so first let's get to low cost indexing uh of course we all know that you know we that dnn compression is a pretty big uh thing these days we can take pretty complex dnns and compress them to make sure that they provide sort of like roughly the same functionality but this end up running far fewer layers and and so forth and and this is considerably so we can just go ahead and say that when these you know frames come in and we get these objects instead of running these expensive cnn in just time and plonking them into an index let's just run these specialized compressed cnns this is considerably cheaper however these chain ends are seep cheap for a reason they're cheap because they're also considerably less accurate than these expensive cnns like take this example where we're running a classifier that says what kind of object is it so what we see here is you know an expensive cnn would sort of like produce the outputs by ranking what it thinks the object is you know starting from most likely to least likely like for instance if you take a certain image and say that hey i think it's a truck then i think it's a moving van and so forth this ordering is not going to be the same for the cheap cnn because the cheap cnn would think that hey it probably looks like a moving van to me so the outputs may not really match if you want to look for the the rank one in terms of what the output is but if you do look at it the best result from the expensive cnn what we've seen empirically is is within the top k results of the cheaper cnn and for a very small value of k so this gives us a great uh you know leverage in terms of how we can use these cheaper cnns and fix the problem of accuracy so instead of generating a simple index let's generate a top k approximate index here where what we do at ingest time is we take these frames we of course run these compressed cnns and then now go ahead and put everything that's in the top k into the index and so now when we put everything in the top k into the index when someone is querying now this is not just an index lookup as we were doing before but now you'll have to get these outputs from the top k index now we'll run it through this expensive cnn but now that we're running it through a far lesser number of frames the expensive cnn has to do far lesser work than before and after that we we return those frames and and so this way we ensure that you know while there is uh uh you know well there is low-cost indexing we also you know we also produce highly accurate results because of the fact that we have expensive the expensive cnn at query time this of course leads us to the next question which is okay it's great but how do we achieve low latency because we've just added a bit of additional work at query time earlier we just had to look up the index now we have to run it through this expensive cnn to get precision so to address this non-trivial work that we have to do at query time what we do is we use clustering based on the extracted features of these dnns uh you know just a little bit of context you know before the fully connected layers in these convolutional networks there is a feature vector that gets produced and and usually what we've seen is that images that have the same feature vector are mostly visually similar as well and this has been shown in multiple studies in terms of you know prior work so essentially what we do here is we take these these uh you know we take these these feature vectors and we cluster them and so so what we do here is in this in this flow that i had before first we cluster the objects and then produce the top k results and then only on the centroids of these clusters do we run these expensive cnns note that the number of centroids are far lesser than what we had otherwise so this is kind of how you know the the whole the the the workflow looks like in terms of both achieving cheap indexing highly accurate results and yet not too much work at query time hence giving us low latency uh of course there are also tradeoffs that you can do between the ingest cost and query latency like for instance what's the cheap cnn that you pick perhaps you can pick a slightly more beefier cheap cnn and then that might give you a lower value of k which means you might end up doing less work at query time so these are things that a system operator can set depending on the choices they make so for our results you know we use the same videos as before the traffic and the surveillance videos and note that we strive for target recall and precision values of 99 percent and so this is kind of what the solution looks like i mean this is where our ingest heavy baseline is and this is where our uh you know a query heavy baseline which is one of the recent work uh from from prior work that we have and so we end up falling here which is actually substantially cheaper than in just heavy baseline and much faster than a query heavy baseline so with our design you know remember this price point before that we have of close to 400 a month per stream we can actually get it down to single digits and likewise for a five hour footage we can do it in a handful of minutes to go through a five hour footage let me show you a demo of the system in action right now okay so here is the demo of focus as we can see we can select different kind of video that we want to query in this prototype system and different class so let's say i want to query this string video that we call sit hard and we want to query a object that's dock because as a ct planner we may want to know how many dots and what kind of dogs show up on this street and we can see that this video it has sometimes people going walking around and bicycles and different objects um so if we don't have focus we just have baseline then we do query we click run then the query will go on um and then as we can see that we need to check many frames basically uh around 10 000 frames before we can get the results which is going to be very slow as we can see on the other hand if we have focus uh we can do the same query but now we select that we use focus uh which generate approximate index and so we can save a lot of query time so now if we click run then we can see that the query latency becomes very fast it's only three seconds and you only need to check uh 100 frames then we can return results we can see that there are two segments and in this segment we see dog and the other one we also see dogs it's pretty cool and we can also do different kind of query for example if we are interested in uh bicycles in a different video and we can select this and we can again do this query again because of the design and focus we only need to check very few frames at query time so we only take two seconds to return the results we can see the bicycles moving around pretty accurate and if we come back to the baseline it's still running it's going to take a much longer time since we don't want to really wait i actually run this query uh before this recording as we can see that uh at the end of this query of baseline we overall need to process more than 10 000 frames and the latency is more than four minutes so compared to focus there's only like a few seconds there is two orders of of magnitude improvement and we can see the resent results are exactly the same so these two segments with dogs so we can see that white focus is a much better option than baseline i also want to say that this is part of the code that we are releasing so youngsha would also be showing a demo of this indexing in action uh of course some of the techniques that i didn't talk about today but we actually quite excited about is you know as i said at some point that uh you know organizations organizations deploy a fleet of cameras together how can we use them together for video analytics uh privacy is a big thing we are we're quite cognizant about so how is it that we can do private video analytics both on the edge and on the cloud and finally this is not all just about video dnn inference but we also you know have projects that go on to update these models on these edge device continuously so to conclude the first part of this talk i had shown how approximation is a very handy tool for us to run video analytics across the edge in the cloud i had shown how we can use this live video analytics as as is to to pipeline well into interactive querying of stored video sets as well so in the next part of the talk i'll hand it over to my colleague yanchow shu who would be explaining about the rocket video analytics platform that we've open source he'll show you how you can get set up with it how you can play around with it and and how you can use it for your own applications thanks ganesh hi everyone my name is ryan chao and i am a senior researcher at microsoft research so i'm gonna walk you through the code of rocket and show you a few configurations of the pipeline so at this point we have some good news to share with you as ganesh mentioned microsoft rocket video analytic platform has been released recently and everyone can visit the url below and clone the repository from github and the link also contains instructions on setting up the environment this rapport puts together a hybrid edge cloud video analytic platform which is built on c-sharp on a car and the cool part is that it allows tensorflow dn model plug-in and gpu and fpga acceleration as well as container docker containerization and interactive querying for the after the fact analysis so many of these components in this platform are key building blocks for video analytic applications including traffic counting near-misses detection vehicle speed estimation crowd control and so forth so functions you may find useful in this code include a line based alerting and a selective dn cost for efficient gpu and fpga usage and it also features edge and cloud partitioning in which you can run cascaded costs to cut cloud dns for example the microsoft cognitive services or azure machine learning or shift it to edge only mode if the network is not available and also we have an interactive after the query after the fact querying on stored videos for example you may want to find all the frames which contains the red cars from the previous week so next i will walk you through the project and introduce five pipelines suited for different real-world scenarios the microsoft visual studio is recommended ide for rocket once you install this visual studio you can simply click vap.sln to launch the project so in visual studio on the right hand side you will be able to see 14 in total projects under the solution vap and each of these projects roughly corresponds to a module in the rocket platform for example there is a project called decoder and in this decoder.cs you will find a function called get next frame which returns the next frame of either live video streams or a pre-stored video file and similarly in the dark net detector you will find a frame diane yolo.cs where there is a function called detect which use darknet yolo object detection model to generate a list of yolo tracking items provided an image or a frame from the video and the main program the cs file is it sits inside this video pipeline core projects so in line 143 you will be able to see a big while loop in which each frame of the video is going through different modules which module this frame is going through is configured in a file called app.config so in this file we have pre-compiled six different configurations for example if we set the value of the pipeline config to 1 it means every single frame of this video will be analyzed by the dark knight yellow v3 model to execute this code there are basically two options you can either do it in visual studio in the debug mode or run and run it in a command like tool like windows powershell before running that in visual studio you need to pass several application arguments those arguments are defined at the very beginning of the program.cs file as you can see here they are in total five arguments you need to pass into the application the first one is a video url so the rocket pipeline accepts both local video files like a mp4 avi file or live video stream like rtsp stream and the second argument is a line configuration file so in rocket project we use line to define an area of inches where you care about this object and the following two factors are sampling factors and resolution factors respectively so the sampling factor determine how many frames you want to run on this video for example if we set the sampling factor to 2 it means we will drop half of the video frames in the file and the resolution factor decided how we gonna downscale this video to speed up the processing and of course you need to specify what are the categories you are interested to be detected in rocket so here we have a sample video put into this repository which is named sample.mp4 and there is also a sample line configuration file corresponding with it and here we set this down sampling rate to 2 which means we drop half of the frame and we don't change the downscaling factor which is set to one by default and we care about all sorts of vehicles including cars minibus minivans and trucks so the first pipeline that we are going to show is a configuration that generates alerts on objects in the area of interest as i mentioned before in rocket we use a line to define the error of interest so this sample video we included in this repository is from a traffic camera in a total of five minutes video there are four cars and one bicyclist moving down the road in this scenario we draw a line near the top right corner of the scene to detect all objects moving downwards this snapshot of this video shows where the line is in the sample line configuration file so in this file each line corresponds to a line you draw on the scene starting with the name of the line the id of the line and four values defining the coordinates of the of this line and the last parameter is an overlapping threshold so now let's go back to visual studio and set this pipeline value to one and run this pipeline and see how it goes after loading yolo models rocket starts processing the video by running dn inference on every other frame as you can see even if we set down the sampling factor to 2 the average processing speed is at around 20 fps which is insufficient for real-time video analytics if you go back to file explorer you will find an output folder named output or in which you will find all the detection results from this dna model as you can see at around frame id 1300 the first white car shows up which triggered this line and captured by this dark night yolo model and 22 alert was generated switching back to this folder you will find all the alerts caused by different generated on different frames since running the dn on the entire five minutes video takes a long time i'm just going to show you the pre-generated result of running yo on this video as you can see all four cars are successfully detected although it may take tens of minutes so to speed up this detection rocket platform also features a background subtraction-based detection module that can be used as an early filter compared with dna inference background subtraction is much cheaper and can be running on cpu at a fairly high fps so now let's switch back to visual studio and see how the fps goes when we adding this background subtraction based detection as you can see the average fps goes as high as 90 fps because at this moment there's no objects so no dn is caught and it's purely running background subtraction on cpu at the time when a car shows up the background subtraction triggers the dn and the dn finds a car and send the result to a cloud database so even considering the latency of sending the data to a cloud database the average fps remains at 60 which is double the speed than real time as the application runs i'm going i'm also going to show you how to run a db query to get the detection result from the cloud so for example this is a simple gui to send this query to the cloud database we have already set up a cloud database as well as azure blob storage to store all the detection images if you run a query to find all cars being detected during this time period you will get the results in half seconds with a total count of 21 so all the images as well as with as well as the metadata of the detection will be fetched immediately without processing tens of hours long of the video similarly all the detections can be found in this output folder output all and the third pipeline we are going to introduce is detection object with cascaded dns so on top of pipeline 2 we are going to add another dnn before calling this heavy dn full-fledged yellow so in many cases we can run a cheaper dm which could already give you a sufficiently high confidence so in that case only if the confidence is not high enough we will not proceed to call this heavy dn so let's set this pi plan value 3 which corresponds to this cascade at the end here two models are loaded which are the tiny yellow v3 and the full-fledged yellow v3 background subtraction based detection keeps running on every other frame at roughly 100 fps if we find some movements on the foreground it will firstly call the tiny yolo model as long as the chip model has a sufficient high detection confidence the heavy model will not be triggered so here in this output folder the output ltdn contains all the detection results from this tiny yolo model here you can see it detects the first white cars and the second black cars while the app is still running the third car and the fourth car will also be detected at a later point so the third car comes and some new up some new detection results are generated and if you switch to this output cascaded dn folder you will see the result of heavy yellow object detection model which as you can see are more precise and the confidence score is much higher so in pipeline 3 we have three modules sitting on the edge however this may not be feasible especially when the heavy model becomes bigger and the number of video streams increases so here in pipeline 4 we move the heavy dn detector to the cloud making the shared service to multiple azure analytic pipelines so in our azure cloud we have created an azure machine learning service and deployed a darker image with tensorflow radnet model the azure machine learning azure machine learning service can take advantage of cloud fpga accelerators and provide grpc and tensorflow serving predict apis of course you can always deploy a cloud model using a customer docker based image so many of you have already been using docker and containers so for those of you who are new to docker let me say a few words about the extremely popular technology so in short containers allow developers to package up an application with all of the parts it needs such as libraries and other dependencies and ship it all out as one package by doing so developer can rest assured that the application will run on any os both linux and windows regardless of customized settings that machines might have that could differ from the machine used for writing and testing the application so for example you don't need to worry about if.net core runtime is installed on the hostel os another difference in pipeline 4 is we switch the local light dn detector from a yolo b3 model to a tensorflow model the rocket is designed to be compatible with different pre-trained tensorflow model here as you can see in the visual studio we use a fast rcn model based on rasnap but you can freely change it to another model from the tensorflow model zoo now let's set the app config value to 5 to run this pipeline 4. as you can see at the beginning the tensorflow model is pre-downloaded to the local device and then the gpu is initialized so you can see when a car shows up rocket continues to call the tensorflow model and once the tensorflow model detects there is a car without sufficient confidence it will proceed to send a remote call to the azure machine learning service to verify whether it is a vehicle of interest so in the output folder you can see the result from the cloud azure machine learning service which successfully detects all this object as a kind of vehicle so the last pipeline that we are going to introduce today is the container plus agent cloud split so different from pipeline 4 here we also containerize the local pipeline of running background subtraction detector and light dn detector in this way this pipeline can be transplanted to a different hosting os for example linux mission and being executed there so here i'm going to show you a remote session of running this pipeline on linux machine sitting in azure i'm not going to show you how to build this docker image because it may takes tens of minutes instead we have already previewed this image and put it down to this linux mission so this is the docker image that we are going to run on around the docker around the rocket pipeline to run this pipeline we simply mount this disk and also similarly to run that in visual studio we supply the videos the line config file and downstairs down sampling factor and down resolution factor so after downloading and extracting the tensorflow model the app is running on the linux mission because it's a cloud device so the cpu is much more powerful than the laptop we are demonstrating here so as you can see as a result the average fps goes as high as 90 fps so you can configure the pipeline to send this result to a cloud database as well as some local storage service and here i'm going to log into this stp into this device and see the outputs in the file system so here as you can see there are also pencil detection results generated in this output our folder so this concludes the webinar these are just five sample pipelines of the rocket platform so you can configure and put together your own pipelines and also reuse the output of the existing pipelines for further analysis of the video content so we hope you get a sense of how in general the techniques look so more materials and update can be found at the rocket project website we look forward to your thoughts questions and bug reports thank you uh hello everybody this is uh ganesh from microsoft and we've been switching over to the the q a part of the the webinar so let me start with uh you know some of the specific questions uh that were asked about the the first part of the the techniques that were presented one was the ability of the profiler or the system to to cope with you know real-time video streams and what is the latency for processing uh each of the video frames so so you know upfront i would want to state that yeah the main you know part of the video analytics platform is to be able to process live video streams which means that if it is a 30 frames per second video stream the budget that we have to process each video frame is about 33 milliseconds so so that is something the framework automatically takes care of uh you know even when we're doing the profiling to find the configurations uh we only keep configurations uh at which we can you know sort of like go at the the required frame rate and there would not be any buffering of the frames as such so for instance if we do have a heavy configuration then we would appropriately make sure that the frame sampling is chosen so that you know there is no buffering of the frames and we keep up with the live frame rate um and then there was another question on the second part of the talk around video indexing and video querying as to uh you know how do we actually store the the the outputs of the cnns that we do during ingest and how do we do the clustering so just to add to the content in the webinar uh we would like to say that you know all the frames that are chosen for ingesting we run it through the cheap cnn uh you know of course there is an optional aspect where we might the ingest itself could have a frame sampler in it but all the frames that are chosen for ingest is run through a cheap cnn and once the outputs of the cheap cnn is available as we saw in the webinar as well there is uh there's both two kinds of outputs one is the actual you know classificatio