Faster dataset annotation and model evaluation with Roboflow - Made with TensorFlow.js

Key Takeaways

in this interview we're joined by Brad who's working on a product that empowers developers to build their own computer vision applications no matter what their skill set or experience may be first of all Brad tell us more about who you are hey Jason thanks for having me uh if I was to describe myself in one word it's a hacker I've basically just been building stuff and putting it online for as long as I can remember um I got my start really in the social gaming sphere uh when I was in college I built a Facebook application that ended up getting over 10 million users and I dropped out of school and ran that for about a decade and that really brought me into the augmented reality and computer vision world uh where uh I started working on board game helper apps so I made a AR Sudoku solver and an AR Boggle solver and kind of launched those and hope that they'd Inspire other developers to create similar things and in talking to some of my friends what I learned was they didn't need inspiration they really needed tooling and so that kind of led to a pivot to working on developer tools to help other people create computer vision powered apps wow that is quite the story I love it that's uh um very very interesting now what exactly is robofloving and how does that work with webml and tensorflow.js yeah so roboflow is a developer tool um basically we're trying to help developers give their software the sense of sight so taking image data and turning that into actual information that you can utilize for cool things um so kind of the key Insight of roboflow is that the way that humans gather information from the world around us is primarily through our eyes that's not really the case for computers yet and so we're trying to change that and put this into the hands of all developers you know not just companies like Google that can spend billions on self-driving cars but all the fun use cases that you can do to just add Vision to your applications as well so what that means in practice is we build an end-to-end computer vision tool chain it's everything that you need to do everything from collecting images to annotating them to creating data sets to training a model to then going out and deploying it into the wild and building an app and we've had over a hundred thousand people that have built on roboflow so far including you know giant companies in fact over two-thirds of the Fortune 100 have used Robo flow for something so far but also thousands and thousands of individual developers and students that are just building cool hackathon projects you know in their basements and in terms of the how we use tensorflow.js there's kind of three ways uh kind of percolates and uh Spider websites it's way through our entire product um so first we use it as a model assisted labeling tool um so once users have trained a model they can then use that as a jump start for improving that model so they get predictions back from their model and when they're annotating images that's their starting point so instead of starting from scratch and having to draw all the boxes really you're just correcting your model and teaching it new things you don't have to teach your model the things that it already knows it's kind of the um the philosophy behind that and then secondly uh we use tensorflow.js to let users preview their models directly in their web browsers which is really magical you know you hit train and the model comes back and you can hit go and you just see it right in your browser running on your webcam you can try your your model in real time which I think will get a chance to preview here in a second and then third we kind of wrap that functionality in a developer SDK and we let them build their own applications that they can deploy to their users browsers to create all sorts of really cool things wow that is truly amazing and a hundred thousand users that's incredible so you might be our largest huge of tensorflow GS5 I've interviewed on the show so far so congratulations on that now I think it's time to show our viewers how they could potentially use this product can we see a demo um so this is an example of how Cardinal Health is using roboflow to improve the back rooms of pharmacies so they have a division of their company um that works with pharmacists where one pharmacist can manage several locations remotely over a video stream and so this runs in the browser on an iPad and previously it was just a video chat between the pharmacist and the technician and then they used roboflow to add on a pill counting feature that helps the pharmacist uh be augmented by the computer vision model so they don't have to count from scratch they can use the model to estimate and then you know adjust up and down where the model has you know not gotten an exact count wow that's really cool I love how this is solving real world problems in Industry as well which is really great to see uh what's next totally uh so let's pivot to another uh industry use case but maybe a little bit more fun one um and this is how Wizards of the Coast is using roboflow to power a feature inside of one of their games so when the pandemic hit people had to switch from playing Magic the Gathering uh in person to online and so uh they acquired this app called spell table that assists uh in like playing twitch remotely and so they added a computer vision feature to their app with roboflow that can identify what the card is that's being played and then the players can hover over the top of it and see the full scale version of that card so they can you know read the fine print even if they their video stream isn't high enough resolution what that's wonderful I'm sure we've got some magic the Gathering players in our viewership today as well so that's awesome there and I love how that's enhancing the digital experience when you you unfortunately remote in these days uh you can really bring that to life in a way that's readable and understandable by all so that's that's really good to see awesome what's up next so now we're going to jump to something uh even more fun which is how the viewers in the audience can utilize this uh right out of the box so this is a website called rubber floor universe that we built as a public space where our users can share their trained models with other users and then those users can build things with those so this is an example of a playing card uh model that one of our users created uh and so you can see that on the robofoil Universe website there's over a hundred thousand data sets and 10 000 pre-trained models that you can view uh this is one of those and then we give you the code for you to implement this directly into your application so this model finds playing cards and then if we can flip over to uh uh the movie Magic I actually did a YouTube live stream of taking this and building an application on top of this so I took the output of the model which told me what the playing cards were and I created a blackjack strategy app that would find Blackjack hands and tell you whether you should hit stay double down or split wow so this is basically Cloud counting essentially so I mean uh yeah if anyone's uh watching this combining us with some AR glasses could be very interesting so um very cool very cool um okay what's the next demo I think you want one more for us yeah uh so now let's uh do a quick demo of label assist which I described earlier as you know how you use your model's output to improve the labeling process and make it so that you can iterate more quickly um so here uh we trained a model to identify cars and then we are adding more uh images to our data set so we just invoke the the model uh and turn it on and then in our browser tensorflow.js is giving us a starting point by which to label and so then you just find where the model mistake and you can teach it new things instead of having to teach it the same things that it already knew all over again very nice and so are you able to be transfer ending in the browser of tensorflow.js fin to like to update the model on the Fly essentially and then you'll then using the new version right there in the browser right is that correct yeah totally you can use that previous checkpoint for transfer learning the training for that actually doesn't happen directly in the browser but it's a human in the loop transfer learning maybe wow so it's really amazing to see that how many different ways you're able to use Robo play for this and of course with the labeling side of things that's super useful to people who are trying to create new models as well like anyone anywhere can then start getting a better model to do the labeling in real time as they're labeling which is really really incredible so great welcome back so I guess at this point why did you use tensorflow.js as particular solution yeah there's actually a ton of reasons that we use tensorflow.js I think primarily uh the the first thing that gravitated us towards us was just what a magical user experience it is being able to just click a button and see a model working on your webcam feed and that's incredibly difficult to do if you can't run something in a web browser you can imagine you know the opposite user experience of like telling users to get a certain graphics card and then install a whole bunch of drivers and like get their system set up and you know it's hours versus seconds uh it's just like night and day difference and using tensorflow.js creates that aha moment where they can really utilize and understand our product really quickly um but besides that like there's a whole bunch of business reasons as well and one of those is cost by running these models inside the user's browser you know we don't have to spin up a whole bunch of gpus in the cloud the users are using their own machines computing power which also means that the imagery never leaves their computers and so you know we never are seeing the output of their webcam feed when they're using these models in the browser that saves on bandwidth that saves us in cost it you know improves privacy it's kind of like a win-win all the way around uh and then there's another thing which is um until actually relatively recently um using GPU accelerated models was really hard outside of tensorflow.js you had to you know have a very specific brand uh graphics card but with tfjs you can you know use whatever your user has whether they have you know whatever brand of graphics card it is or even if they only have a CPU it just kind of works and so that allows us to fulfill our mission of democratizing computer vision and getting this stuff in front of as many developers as we possibly can yeah but these are really good answers and I'm sure given the scale of your product of hundreds of thousands of views using it but saving quite a bit of money per month if you were to use server-side graphics cards to deal those inferences and things like that right so yeah I mean it wouldn't even be possible and you know the reason that we can offer so much of our product for free is because you know we're able to offload that onto the user's machine and the more users we have have it isn't really costing us a ton more money and so that you know doubles as a way that we can get this out in front of hundreds of thousands of people instead of just a few hundreds yeah and English I mentioned of course privacy as well and given that the medical applications you shared us earlier on with the counting pills and things that you know there's obvious reasons for privacy in some of these medical situations too so I guess it's a win-win there for you guys so if that's really awesome to see um so it seems like it's a great fit now if folk wants to try this for themselves how can they do that yeah so the first thing to do would be explore Robo full Universe like I mentioned uh there's over a hundred thousand data sets and over ten thousand pre-trained models uh for basically every use case that you can imagine there and all of those models you can actually try in your web browser that makes a lot more sense for the ones that you know are uh detecting whether you have a face mask on or not or playing cards then it maybe does for some of the manufacturing use cases but it should give you a you know a great um idea and launch point for starting your own projects uh and then secondly there's a whole bunch of projects on GitHub that go One Step Beyond that and show you how you can take one of those projects and build something so we showed earlier the Blackjack example another example is an app that I actually built last week that helps you parse your drone video and I trained a model to find solar panels and then it plots those GPS locations on the map and so we actually have dozens and dozens of aerial models on riboflow Universe already and so you know it's not just solar panels that you can plot you can also find things like swimming pools or you could use this for search and rescue to find you know people who have gotten lost in the woods or for monitoring forest fires or any of these use cases it's kind of like this generic technology and there's an open source repo that you can get started with that's awesome I'm a big fan of drones myself so definitely very interesting to me and um you know I guess one extra question here where do you see this all going in the future uh I mean our hope is that computer vision becomes a tool in every developer's tool chest you know we've reached 100 000 developers so far but there are millions and millions of developers out there that haven't tried computer vision yet and so we make a core product that is free for you to sign up and create your own custom computer vision models that then you can utilize inside of tensorflow.js on the technical side of things right now we support object detection with tfjs but we're working on semantic segmentation and instant segmentation and classification and some of these other problem types that will let you do even more in your user's web browsers so stay tuned for that awesome and just to clarify as well um we were speaking about many of the data sets you've got available on web people to use are they all reusable by anyone or some of them come with like licensing restrictions or how does that work exactly yeah so our business model is if you're willing to open source your projects then we give you the product for free and so most of our users opt into sharing their projects publicly and we know that's not you know feasible for a lot of businesses that think of their data as like a proprietary asset and so that's where our paid product comes in of it's kind of the GitHub model where you know if you need private data that's what the paid product is for and if you're willing to open source it uh then it's free and so that's how there's you know hundreds of thousands of data sets out there available for people to start with awesome well that's really wonderful work and of course for various watching let us know in the comments below how you might use tools like this to bring your ideas to life in your industry thanks Brad again for joining the show today and I look forward to seeing even more tensorflow GS embrace your platform thanks so much for having me Jason it's a pleasure [Music] thank you