Monster Apps: Meatscope + Polymon (Polymer Summit 2016)

Key Takeaways

[Music] hello everybody I'm Chris Joel I uh built polyon and meat scope and uh today we're going to talk about monster apps so a few months back the polymer team flew to Sydney to hang out with some of the Chrome developers who like to work upside down and you may have known this about Australians I didn't really know it but Australians are really great collabor they love to collaborate I would get tired of collaborating and I would turn around and these guys would be uh you know looking at me expectantly acting like we were going to collaborate some more and I just really couldn't take it I have to admit that a certain point I had collaborated so much that things just cross the threshold and I don't know things got a fuzzy and then this happen what you want entertain don't do it man don't do it g g g greas watch out when you collaborate it's it's dangerous so as I sank into the darkness into the cool Sydney Coastline that evening I had a few moments to reflect what I been doing with my life had I gotten so caught up in this collaboration that I'd forgotten why I got into web development to begin with I wanted to change the world I wanted to build apps that would touch people's lives that would make the world a better place thanks to this moment of introspection I had the inspiration I needed selfie gifts the internet needs selfie gifts I mean everybody loves selfie gifts they combine the two best things on the internet selfies and gifts so I did the next thing that any self-respecting web developer would do I went out and I built a web app okay actually the first thing I did was I registered a really cool domain name I called it meet scope uh I registered meet scope. camera in case you didn't know you can get a camera domain then the next thing I did was I went and built a web app but when I first got started I wasn't really sure if this was something I could even do I mean I needed to get access to a camera I needed to process image data from that camera I needed to store that image data in in my web app was this stuff that could even be done I mean this is supposed to be a phone app can phones even do this on the web well this is a story about how the web platform had my back and how I used the web platform to deliver selfie gifts to everyone so we start with the media capture and streams API this is a really cool API that's available uh in pretty much every browser that lets you access uh input from camera input from microphone and record it uh or display it to the user well okay it's not quite available in every single browser but we'll come back to that in a minute so what does it look like it's a really great API it's got all these great promisify methods it's most of them are hanging off of Navigator media devices here you can see I'm calling enumerate devices and this is basically a promis promisify API that gives me a list of all of the input devices available to the user and here you can see I'm actually filtering by Dei uh device. kind that matches some kind of video string uh and this is how I get my list of cameras uh another really useful method in this API is get user media so using get user media I can select in a fuzzy way or precise way exactly what I want access to and uh it returns a promise that resolves with a stream of input from that device uh here I'm specifically requesting access to one camera and when you put it together with the HTML video element you get something cool you essentially get a video element that's displaying stuff coming out of a camera on your device now the HTML video element is cool for two reasons one it lets your users see exactly what's going on what you're taking uh image images of the other reason why it's really cool is you can pair it with canvas draw it to a canvas and get image data out of it so here I'm taking a canvas I'm measuring the width and the height of the video I'm drawing the video to the canvas and then I'm using a method called Draw image draw image is a little misleading it says draw image but you can actually pass a handful of other elements to the draw image method uh and also the dimensions the width and the height and then voila there's pixel data all right here's a demo so on the bottom right I've got a HTML 5 uh video element that's showing stuff for my webcam and in the back I have a canvas so if I need to make my selfie face I can go like this and you guys can stare at that for a few minutes um and so what are we doing here we're using the web web platform to capture image data from the webcam and if you're lazy like me and you like web components I've wrapped a lot of this up in a series of web components I call meat scope elements uh so from top to bottom uh I've got meat scope devices which basically accesses the enumerated devices method and filters for all the cameras and reveals one of the cameras as selected camera I'm binding that to me scope user media which takes in a device and then spits out a stream and then the stream gets bound to Meats scope video and me scope video just displays whatever is in that stream in a full bleed video on the device or in some smaller container if you prefer uh so the next thing I needed was a GI encoder and thanks to the vibrant JavaScript ecosystem there's like a million GIF encoders but I picked this one it's really great pretty simple to use uh it looks something like this I create a new instance of a gif I configure it a little bit and then for all of the frames that I've captured for my GIF I add it into the uh GIF using the ad frame method and then I call gif. render pretty simple and in order to actually make a gif you have to record multiple frames and then add them in in sequence so it looks something like this here I've got several canvases next to each other and each one has recorded some picture of me doing something one and then you know you add them all together and it makes it look like I'm going like so each frame when I'm recording a gif I do something like this I draw the video to a canvas context and then I push the image data from the canvas context onto a frame array this turned out to be really really bad basically during a very sensitive time in my user experience when I was trying to record gif image data I was spending 50 milliseconds just reading out of a canvas you can see that most of the time is spent at the bottom there where I'm calling get image data so that was bad but the worst thing was at the very end you see at this red box on the left was me recording the GIF on the right is 40 seconds I spent rendering the GIF and this is all happening on the main thread this was actually on desktop went in my first pass and frankly it was totally unusable it's like you record the GIF you drop a bunch of frames and then your user has to sit around for 40 seconds waiting for a bad gift to render and and meanwhile they can't even use the app because the main thread is blocked so what do we do how do we make this better Mya I don't know why that keeps happening so I started to think about you know what could I do to optimize this whole experience of recording gifts um and the first thing I thought of was you know when I get the final GIF I've scaled it down pretty significantly compared to the source data so what if I could do that in advance instead of trying to pass a bunch of Big Image data down to a gif encoder and have it do the scaling itself um it turns out that the canvas 2D context is able to do this for us um the draw image uh method will allow you to pass uh Target dimensions for drawing the video so here I am doing the same thing I did before except I'm passing in the scal dimensions that I want to draw on into the canvas now this is nice because it means I want I go to read the image data off the canvas I'm going to be reading far fewer bytes than I was before but I still thought I could do a little bit better every single time I was recording a frame I was actually doing something like this I was writing stuff to a canvas and I was reading it back than writing it and then reading it back could I do better than that uh well as a matter of fact I came up with a nice strategy that I think balanced uh you know reading and writing pretty well so it looks something like this so what's going on here is it's the same demo as before I just have one canvas and I have one video input and as I'm recording frames each time I increment a counter and I know I need to draw the next frame a little bit offset from the first one and it creates kind of a film strip effect now the result of this was pretty awesome this thing really wants me to print literally a 100 times better so remember before it was about 50 milliseconds per frame now it's half a millisecond per frame because I'm not actually reading any of that canvas data anymore this is so much better because it means I'm not going to be dropping frames while I'm recording a gif also I had this problem still right I'm I get to the very end of the recording and it still takes almost a minute for me to get my rendered GIF the trick here is you really have to keep work off the main thread it's not okay for you to lock up your user experience for almost a minute uh just to uh you know do some heavy duty work so how do we take care of this Well turns out the platform has another solution for us web workers so most of you probably know what web workers are how do we use them here here I'm creating a new uh web worker this is just a basic web worker and I'm passing in my GI en coding library and then I'm creating a message Channel I like message channels because they're kind of like a two-way walkie-talkie between the main thread and the worker thread I pass the port from the message channel into the worker and then I listen for messages on that Port so that I can do communication now if that looks a little verbose the platform actually has a really cool kind of worker called shared worker shared worker basically works the same way as a normal worker uh with a few exceptions so a shared worker when you create a new instance of it if an instance has already been created it will share the same worker thread with the new instantiator of a shared worker also shared worker comes with a message Channel kind of built in so when you get your worker instance it already has a port that you can communicate to the worker with now shared worker isn't available everywhere and in fact it's possibly on the chopping block for the web platform but it's a pretty nice uh um class to use and if you like it like I do uh we've built a class called common worker for app storage um which works almost exactly the same way as shared worker so you can create a common worker you'll get a worker instance back you'll have a port that you can use to talk to the worker and um excuse me and it also shares the property of shared worker where it only creates one worker thread for every no matter how many common worker instances you've created now inside the worker what we do is we import our GIF encoder we listen for messages on the port uh that we get from the client every time the client instantiates a new common worker we encode our GIF and then we post the GIF back to the main thread it's that simple this was an awesome Improvement moving stuff off the main thread me meant I went from 40 seconds of encode time down to about 100 milliseconds now if you look at that and say well 100 milliseconds is actually still quite a bit of work it turns out there are some hard limitations to what you can do with HTML canvas if you get a bunch of image data out of a sufficiently large canvas you're going to incur some cost but I'll take 100 milliseconds over 40 milliseconds any day of the or sorry 40 seconds any day of the week um it's also worth noting that uh there are standards at work to get worker or sorry canvas that you can use off of the main thread so look forward to that coming in in the web platform soon so this is an example of what I got from all my performance optimizations here I'm recording multiple gifts one after another each time I record a gift I start another one and those gifts go and they encode off the main thread and meanwhile the main thread is nice running 60 frames per second everything's good so your users when they're recording their very important selfie gifts can end up with results like print dialogue no this [Applause] [Applause] so I have my selfie gifts but what do I do with them I mean I don't want to just throw them away well it turns out there's an element for that at Google iio 2016 we released a bunch of experimental elements that we call app storage so app storage is kind of based on this idea that spawned out of Firebase elements uh where we saw the value of creating declarative elements that give you access to storage layers all right so what do they look like roughly they look like this uh we have documents and we have queries documents reference specific Blobs of data by ID queries let you construct uh ordered limited queries so that you get lists of data and iterate over them app storage elements let you synchronize storage and state in your app within the app and also between your app and the cloud if that's something that you want but not necessarily and today we have two bodies of app storage elements that you can use many of you are probably familiar with polymer fire there's an awesome talk by Michael BL BL earlier today uh some of you might not be familiar with the fact that we have a series of app storage elements built around pouch DB so I really like uh pouch DB a lot it's nice because it lets me have access to index DB which is a piece of the web platform that is available whether you're online or offline it's a database that's right there in your browser and I really like to use the web platform so I ended up choosing pouchdb for meat scope now what does using pouchdb even look like well in JavaScript it's something like this you create a new instance of pouchdb and pass in a name for your database and anytime you want to store data in it I mean that's really all you have to do anytime you want toore data in it you just call db. put and pass in some object and you get to store that in pouch DB in index DB in your browser uh and you can see in this example I'm actually storing binary data because index DB can store binary data which means I can just throw all of my uh gifts into pouch DB now I just want to stop and talk about how awesome this is all right it's 2016 I have a database in my browser I can store structured objects I can store binary data I can just throw this stuff into pouch DB and forget about it I mean it has virtually unlimited space so this was really exciting for me but what does it look like to use it in practice with polymer so here's pouchdb query here you can see I'm specifying that my database is me scope I'm create I'm crafting a Lector and sorting and specifying fields that I want and then what I get as a result is gifts now gifts is just an array of rows I can pass that into a Dom repeat iterate over it and just spit out my gifts it's really that easy and if I want to interact with a document directly I have pouchdb document so I'm binding it to a specific uh document with just some random doc ID and and the result is I get GIF out of the data property and with GIF I can just make changes and buy a two-way data binding all of those changes are going to transparently reflect in the storage layer so you know a bit about selfie gifts it's time to take selfie all right I'm going to need your help for this everybody say web components all right so just so you guys can see I've got my phone here it's encoding the GIF over there in the top left but just to prove that I'm not full of hot air I'm G to take another selfie yeah I'm looking good up here on stage all right there's the gift that we recorded and oh and it's frozen well you guys get the [Applause] idea and enough about selfie gifts let's talk about important things like coffee so a few months back uh Justin from the polymer team and I were sitting in a coffee shop and uh you know we were looking ahead to the polymer Summit we were like a little anxious cuz you know polymer 2.0 was you know coming about and we weren't sure what we were going to talk about and we had a conversation that went something like this Justin's like hey we should build a really cool app for our users at the polymer Summit to show off how cool the web platform is and like get them engaged get them interacting with the team members don't you think that'd be cool and I was like yeah yeah yeah let's build a video game that's a great idea you have so many good good ideas Justin and then Justin was like no I mean we only have like three months games are hard uh what about like a survey or something something where they can like fill it out and then like interact with each other somehow I don't know and I was like Hey guys guys Justin wants to build a video game tell him about it Justin it's gonna be awesome and Justin didn't really respond to that he just sort of disappeared so uh anyway that was the conversation that on meat or sorry polyon and most of you are familiar with polyon now polyon is a location-based monster catching game with player versus player battles now I just want to stop for a second and say awesome this is on the web platform you can build this on the web platform please build things like this on the web platform yes clap that's [Applause] awesome so Justin had something about you know he had a good uh intuition about this like building a game is not easy uh it can take millions of dollars to build really cool video games and often they go over budget and over time how was I going to really prove that this was a thing like I had to have a really great minimum viable product and I had to have it fast otherwise this was probably not feasible for the polymer Summit timeline I mean this was August it was mid August when we started doing this uh and so I needed to come up with a cool way to get people interacting with each other I need to be able to rely on the web platform to do it so I looked a little bit into physical web now physical web is cool and all but you know maybe it's a little too cool too Cutting Edge not everybody has access to it uh I also looked a little bit at web Bluetooth so web Bluetooth is a series of awesome new apis that are landing uh and I wanted to be able to scan Bluetooth beacons from JavaScript and get access to all of them so I could see what polyon were nearby unfortunately those are kind of still on flight and did you guys know that there's like a web NFC API I didn't know about this but it sounds really cool I'd love to be able to catch a polyon just by tapping my phone to somebody else's phone uh that's not really there yet then I stopped for a second and I thought you know what Scott Jensen would do Scott Jensen would use a QR code [Applause] yeah so everybody loves QR codes as we all know and I thought everybody can scan QR codes so let's just use QR codes this would be great and wait a second I want to use the web platform didn't I build a series of components for accessing the camera and processing image data on the camera I think there's something to this maybe I'll just take the meat scope elements and pair them with I don't know a QR code reader Library thing and all of a sudden I've got a QR code scanner on the device and or sorry in the web platform and within a couple of days I actually had a demo that looked like this this is a very early version oh yeah hey look there's a polyon yeah check that out that's cool look there's me in a cafe and oh yeah QR code awesome all right it's okay you can you can clap on my prototype it's awesome [Music] thanks so remember this I said we'd come back to this so unfortunately you cannot access the webcam directly in Safari they just have not implemented the user media API I encourage you all to go to the webkit bug tracker and star the bug to implement this API because it's awesome and fun to use um but I still needed to access Safari and a few weeks into the project Monica came to me and she was like I got it I know how you're going to do it I know how you're going to get access to the webcam on Safari and it turns out that there is an input element that you can use to get access to the webcam you just structure an input element like this uh type equals file capture equals camera accept image/ star and iOS will helpfully show a little dialogue anytime the user clicks on it to take a picture with their camera and and voila we have the ability to scan QR codes in a browser that does not even support the user media API um and this is the demo that resulted so here I am oh well no webcam image but oh look at that I can take a photo oh yeah now many of you have come up to me and scanned my QR code with Safari and you know it's not perfect but I'll tell you what Progressive enhancement is not about perfection it's about making sure that every single browser has the ability to experience the app you the way you intended and if it works I think that that's a good thing better than not being able to do it at all yeah please so this is my uh obligatory purple slide otherwise known as the lazy slide in my deck how many of you guys and girls know what purple means I know youall are lying because you didn't laugh at my lazy joke um but this is basically what first paint looked like uh around about the time that polyon became feature complete uh it was about 1.25 seconds in on desktop now this isn't really great I mean it's okay uh but frankly I wasn't doing any kind of purpley optimizations and there was probably a lot of head room for improvement so what did I do uh this is roughly what polyon looks like uh at the top level I have a polyon app component and inside of there I have a series of routes this is just a few of them but you can see I have a start route I have a map route and I have a polydex route uh and for those of you who have used polyon you know that the uh on the start route and on the map route you see the start page and the map at the same time uh here I have the elements that correspond to those routes so I have a start screen I have a map screen I have a polydex screen um and so these the activeness of these screens corresponds to whether or not the routes match uh so what I started to do was I decorated my app routes with declarative information about what element definitions needed to be loaded in order for those routes to be visible so I added some data attributes here you can see I'm loading the polyon start screen and the map screen every time the start screen is active oh boy that's getting old and then and then on the map screen I have something very similar but since the map screen doesn't have to show the start screen at the same time I can just load the map screen by itself and this is just an example of a few of the routes on my app and then I built this thing called The polyon Lazy loader which sounds fancy but it's really not all it does is it observes changes to route and then it matches those changes against the app routes that I've also declaratively added as children to the polyon lazy loader and each time the route matches it loads the data fragments if they haven't been loaded before now just as a lesson for everybody about how important it is to do code splitting and lazy loading I got my first paint time down to 275 milliseconds just by doing this just by lazy loading all the different parts of the app this is really important it's one second of time that you just save somebody on desktop but probably several seconds of time on mobile uh for now thanks a lot for checking out my talk this is Monster apps I'm Chris Joel work on the polymer team it's okay you can catch me on uh GitHub and on [Music] Twitter