Media - #AskChrome

Key Takeaways

[Music] hi welcome to ask Chrome the Q&A show where you get to ask questions of the chrome team and we get to ask them my name is Tom Greenaway and I'm the game's lead for the web developer relations team here at Google and I'm joined by francois before Hey France wires are expert on all things media in the team and today's topic you may have guessed it is media by which I mean video and audio api's on the world wide web the first topic that we're going to dive into is autoplay now the questions we have here are how does Chrome set the media engagement score for autoplay permission by origin and can it be set at a subdirectory level this comes from Jack Palmer on Twitter with the Twitter handle you same Blut so I think the best way to start this question break it down is by explaining what autoplay is so autoplay is this idea that you arrive on a website and maybe it has an audio or video element and that element just starts playing audio so it's pretty pretty straightforward you may have been able to guess it from the name autoplay I think then the natural extension of that is to understand what is autoplay blocking do you want explain that francois sure so back in 2018 we change how chrome angles autoplay the goal was to improve one specific user media experience issue with all-expense in the past we browse from website to website looking for new stuff learning new thing and at some point out of nowhere our speaker starts to yell at us because one random web site decided to play some audio without telling us yeah I really hate that I do as well so that's why we've made those changes the new rules are pretty simple muted autoplay is always allowed meaning that website can still autoplay video without any sound thanks to the muted and autoplay attributes on the other end autoplay we've sound is only allowed and there's some conditions the first one is if the user has started to engage with the website let's say with a button click for instance in that case autoplay wave sound will be as it's a good sign that the study that the user has studied to engage right so that interaction shows sort of like a love of trust from the user towards the website that's interesting because I think there's an exception actually also for pwace right see this is PWA means a progressive web web application which means like a website that's embracing more like application level kind of technologies so a user can actually take a PWA and install it or add it to the home screen of their device then PWS get a special exception from this autoplay blocking right exactly yeah okay like pw8 do not get autoplay blocking basically yeah yeah because that's probably a very strong sign that the user trusts that website and it really likes it okay well back to the questions then part of that also was what is the meteor engagement scores so again and we actually worked on a blog post last year about this if I remember right the media engagement score is this sort of score that we that chrome tracks for websites and we consult that score to see whether or not we believe that the user has demonstrate demonstrated trust towards that particular website exactly yeah do you want to explain how that's actually calculator first of all the chrome media engagement score is not a hidden thing like you can actually go right now to the internal page chrome media engagement to get a look at what's happening under the hood now chrome current approach is a ratio of visits or sessions to the number of significant media playbacks and what you need to know about significant media Rebeck is that the user media consumption must be at least seven second that audio must be present and admitted that the tab must be active and finally that the size of the video must be at least 200 by 140 pixels and from that chrome calculate what we call a media engagement score which is highest on website where media is paid on a regular basis I say yeah and when this score or when the special disc rust autoplay we've sung will always be allowed okay right so it's a so that the chrome now believes that there is a strong sign of trust between division in that web and yeah that's really interesting about the minimum video size that that makes sense I think another thing that's important to mention though is that you can clear like when you clear your history in Chrome that also clears the score right right so that's important from a privacy perspective well also when the question was another topic about the subdirectory level and how that operates for the blocking and the autoplay so basically the short answer is no it cannot be set at the subject cherry level on the web the identity card of a website is the origin and that's what we use for all privileges associated with a website such as permission octa-pie blocking and we have no plans to change that for what it's worth this is a known issue that some website may face today when they share their origin with let's say a video streaming service and a news website sadly there is no programmatic way to solve this for now we have thought about having some virtual origin on the web but that ID didn't make a lot of progress so far if that happens though we could use that for media engagement okay cool one thing I do have one trick to share about autoplay if you are interested about of course today user can actually change the autoplay policy so it's simple let they simply have to actually go to a website click on the lock icon and simply mark the website as being allowed to make some sound in that case autoplay with some will be allowed so I think we can move on to our next topic then which is picture-in-picture these questions come from Rob Patton questions are can we use picture-in-picture for audio playback and can we customize the playback controls in the picture in picture window so francois the let's start with the basics like what is picture in picture or pip yep so this browser features like picture and picture allows users to actually watch a video in a floating window that is always on top of other window so that they can keep an eye on what they're watching while interacting with other sites or applications it was meant for video indeed now the web is full of awesome primitives that when used together can create some magic and that's exactly what we for to have a beep window for audio playback only I see so this sounds like something to be really great for like podcast or music players right so that means you can be you know on a podcast or music player website and you can pop out controls and so even though it's just audio you can control the the playback even when you're multitasking exactly so let me show you how this would work first you would fetch and decode an image like you would normally do then create a canvas element and fill its 2d context with all the image pixels that have been decoded now you create a video element and assign it source to the result of canvas dot capture string and finally by simply setting the mutti attribute to true the video can be played automatically so that then when video that requests picture and picture is called a pip window with a still image gets presented to the user now this is not the full story because right now this snippet of code doesn't allow users to navigate through all available tracks and for that we need the media station API so with the musician by simply setting some media session action handlers for previous track and next tracks the user will get two new buttons in the pit window and can click them to listen to the previous or the next track and that's basically it oh by the way this is what users actually get on Spotify dot-com with the media player right now I see ok cool and what about like custom controls so that would be great in they like we can think of a custom hangar button or a full screen or even a like button that is suddenly not supported today hopefully one day we will solve this issue with what we call picture and picture for arbitrary content that would allow web developers to put any HTML element in a picture and picture window not just a video element hopefully one day we'll get there it is not supported today and if developers like really wanted is there a path for them to I would recommend them to actually go to the github repo story for that please do that ok : file like an issue yeah yeah well star it yeah that's pretty cool I mean that's a great example of how the web of all's right so I mean all software does that so if the beauty of software but I think in particular the web is more democratic in that nature definitely on the topic of evolving api's let's move on to the next subject which is wakelock so we were asked if we could help explain the wakelock api and why it's needed plus related to this topic is a question about background audio to which i think connects i'll start by explaining wakelock so this is the idea that the web browser decides to prevent the operating system from auto-dimming the screen or turning it off completely because the browser intentionally keeps the screen active without user interaction but why is this needed and how does it relate to media so sure so first of all this is very really key to media actually because when you watch a movie or a video on the web you usually press play it and sit back and watch right so now if the video is a film or a TV show for instance then it's going to go for 30 minutes or even two hours and as a user you don't want the device to deem or switch off the screen right right yeah that makes sense the only alternative would be like for the user to have to interact with the device continually and keep the device awake so that would be quite frustrating right so instead right now web browser detects when a video is playing on the active tab and they will tell the operating system to not lock the screen in other words they keep the screen awake hence the name wakelock and that makes absolutely perfect sense for media but i think there is like a new dedicated api for wake lock right i think it's an origin trial it is actually yeah yeah and that's actually like separate to video and audio because obviously we've already got the video and audio elements working the way you're talking about with the interaction with the screen you know and the operating system so this new api will allow developers to directly request wakelock why is this needed you know outside of video and media so the reason is that there are other use cases where developers need this behavior outside of media let's say for instance that you are baking some get cookies from a recipe website and that every 30 seconds so your phone screens turns off how would you prevent that except from maybe touching your device screen with your nose except that your hands are full of flour right right yeah now that's actually a really good use case it's definitely one way at the developer needs to keep the screen awake and I actually feel like though I have used websites that already do this maybe it was like a recipe website I'm not sure but I don't think there was any video audio playing so I'm trying to think like how are they doing Oh actually wait I think I can probably guess but I'll ask you I explained like how are they doing that if that if the wakelock api isn't actually out yet actually what we found is that many developers are placing hidden videos in the page and playing them in the background okay yeah so that's probably not very efficient no not at all okay so this this explains how things have evolved then and why we're building this API because we want to enable developers to just bypass this hack because obviously they're you know they need this feature and they're working around it at the moment and hopefully this would be much cleaner and more efficient got it okay again this is an example of the web evolving developers leveraged an existing API maybe not on the best way to achieve something they wanted out of the web platform and now the web community is standardizing this as a new API and giving the people what they want bingo another related question is and so it's related because it's about like the browser and operating systems if a tab is playing audio and the user focuses on another tab or multitasks to another app are there any chances that the operating system or the browser will claim memory and end the process I know on desktop if I play audio and I change tab Chrome never seems to kill the audio process but what about mobile so you're right about desktop that's what happens but it's quite different on Android if all you explain Chrome will mark itself as a program process which will make it approach if system saying if the system is running into a high memory pressure situation then chrome may still be killed if everything is normal it shouldn't happen now if audio playback is paused on the other end Chrome will act as another Android application and it may be kill as well yeah interesting it's really amazing the thing about all the work that the web browser does to manage you know the developer needs the operating system and the user needs well by the way that's why we call a browser or user agent mmm interesting so those were the three main topics we had to cover francois but i do have a couple of bonus questions so firstly i'd like to ask you what is your favorite web media API and why oh so even though i worked a lot on picture-in-picture I think my favorite API is the media station API we run it all our media notification would be blank and sad by simply providing metadata for the media your web app is playing users will get customized media notification that includes the title the name of the artist of the album and also a nice artwork and that's only five lines of code and as we've seen before with fiction picture as well it also allows web developers to handle media events such as track changing or seeking that may come from a notification or even a hardware key let's say for instance a user hits the next track cubot button damn that will be handled by a media station action handler by the way this is also what powers the new media hub we've launched in chrome recently so if your website shows a blank notification without any controls today please do me a favor and go add those lines of code what about you do you have any favorite API yeah I mean it's not really a specific API but one of the features I really love about web browsers is the ability to have programmatic subtitles I think this is really good for accessibility and for me personally because I'm actually learning French at the moment or perhaps I could say as your profile same antenor mm-hmm it's very useful because I can be like listening to the French audio and also watching like the French subtitles and that really helps you know connect me to okay so moving forward though we have one last question that comes from Ashley gulling on Twitter the questions are what's next for the media on the web and what features are on your radar and what are you thinking of doing in future so the chrome media team is working hard on maintaining and optimizing the current scope of media api's but I'm quite sure as me you're looking for cool new stuff for 2020 yeah so I'd like to mention one thing in particular web codecs what collects hemp's to provide low-level and efficient access to built-in media encoders and decoders the goal here is to expose all the codecs the browser currently used for the HTML media element for instance this will hopefully enable better support for latency sensitive game streaming client side effect all transcoding mmm nice okay yeah I mean as the game's expert I noticed you mentioned that latency sensitive game streaming that makes perfect sense right because earlier forms of video streaming weren't really related to bi-directional communication but obviously for a video game it's streaming in the cloud it sends the video down then there uses interactions need to be sent back up and so that means that the latency is just extraordinarily important because you just cannot buffer exactly okay Francois that's all our questions this was super fun I'd love um we're finished but a quick note for everyone watching if you have any follow-up questions you can ask them to us on Twitter with the hashtag ask Chrome or you can post them on the YouTube video and we'll try and cover them either on Twitter or in a follow-up episode so yeah thanks again for watching see ya ciao [Music]