Eric Charles - Jupyter Real Time Collaboration| JupyterCon 2020

Key Takeaways

hi this is eric charles and today i'm going to talk to you about jupiter real-time collaboration this is about google docs for jupiter notebooks so we want to offer to the user similar experience as google docs when you type something another colleague or someone else should be able to see what you do in real time i'm gonna use quite often the rtc acronym which stands for real-time collaboration and i'm gonna first review the rtc landscape with the different algorithms the specific jupiter requirement we want to address then i'm gonna review the existing implementation we have in the jupiter rtc repository or it is organized with the specification we build on top the architecture and the integration into jupiter lab then i'm gonna follow up with the work in progress we have around graphql or we want to re-layer on top of existing libraries and the work we want to address to be performant before jumping into the content let me introduce myself a bit more so i am eric charles i am the founder of data layer and committer for jupiter app jupiter server and jupiter rtc project i also partnered with coincide for different open source jupiter projects happy to be here with you today to review first of all the landscape we we have around rtc which is quite broad i mean i discussed uh i introduced you with a google docs this is a tool i have used to to build that slide but maybe you know also microsoft live share live share is an extension you install into visual studio code and it allows you to collaborate with someone remote uh when you write some code some buy some files or whatever you can share your content in life with someone those are close solutions you also have open source solutions and if you go back to 2009 maybe you you you have seen google wave this is like the first uh first concretization of uh google moving to collaboration with uh a mail which is a bit more like a chat and so on that open source project have been moved to apache and it is not now retired so it's no more maintenance anymore but it was something very very strong which lead to which has led google to maybe something like google drive you can also uh maybe know uh acandy icmd is a solution where you you write a marked on file and you share that file with someone this is available in a github repository you can clone that repository and and have your own icmd server if you want to install an editor in your application you have a lot of options uh you have something like rear mirror which is a a collaborative editor but a lot of other options are available and if you don't want to build your own application and still today you want to use jupiter notebooks in collaborative way uh there is coca-cola which is uh there since in some time it does not use the jupiter rtc project but it's it's an early adapter of that collaboration feature let's move on to the algorithms so there is a lot to tell i will not have enough time to cover even uh uh in surface what i should maybe uh cover so i will invite you to to go to the algorithm page of our website which is jupiter rtc read the docs dot io and there you will see that we have defined three categories of algorithms the first category is ot operational transformation it is there since a lot i mean a long time maybe 20 years uh it is the one which has been used by google wave 10 years ago and which still powers a google drive you have some open source implementation using tinymc the an editor in a distributed database and the characteristic of ot is that it needs a central server we have a second family of algorithm which is called crdt it is a conflict-free replicated data type so it means that you have generic distributed data types like list and so on and in that case with crtt you do not need a central server so it's more like peer-to-peer you have a lot of libraries available lumino is the one we use the jupiter rtc it's part of jupiter lab you have yjs you have auto merge and so on and finally finally we have a third category of algorithm which is a divs algorithm so it relies on a classical diff between document so you can see the div and it sends that div to a server which does a match patch process it is the one which is used by cokark a few words about the characteristics of those algorithms so they need to be convergent so the convergence of the the output must be unsure if someone type if a user type something and another user type another another change at the end of the day all the user must have the same output so the different inputs must converge to the same output for every user this is what is called convergence still this does not ensure that the output is the one which is intended so maybe the intent of the user is not respected it's converged but it does not converge to the correct output so we need a second characteristic which is intention preservation and still when you review those algorithm you may see that there are some edge case where maybe this user type uh the same thing at the different place or the network is is not real [Music] every time so maybe the algorithm is not uh strong enough it's not robust to those hk's and still they fail this is why maybe some other initiative come with a mix of those ids a mix of those algorithm and maybe something like microsoft read which has been released a few weeks or months ago is uh is there it's like a mix between ot and crdt a bit more details about ot for example so once again please go to the website jupiter rtc read the docs algorithm the ot section we have plenty of links and one of those links is the wikipedia page where you have like a scenario there where two users have the abc sentence or abc string sorry and the first user wants to insert a x at the beginning of that string so at position zero the second user wants to delete a character at position two so he wants to delete c and those mesh messages go and to the server they flow after to the end user and the ot algorithm implementation needs to have that conversion and user intent ot is more like a family of of algorithm while you have many different implementation uh historically the first one was not robust and uh those algorithms have been enhanced and much and much better in time and to assess the oh the algorithm good is you you have those characteristics which is conversion properties with cp1 tp1 cp2 tp2 you have also if you look at all the algorithm manages the timestamp because every user has to communicate the time when the change has been done different methods exist one aspect which is not tribal is all multiple server can can play into the picture because the idea with ot is that all the transformations of the user are sent to a single server which does with ensure that the parameter of the operation are are transformed or adjusted based on the previous uh transformation and then the the result is sent to to all the user to maintain the document consistency and with multiple users that's not that easy so if you want to play more i invite you to go to that website which is operationaltransformation.github.io there is source code available in github also it's a react based application but the nice thing is that you can type a sentence a string and you can play like sending those messages across the network from client to server you can uh clicking on the arrows the black arrow the grey arrow you can play and see all the messages are uh are will impact the server and will impact the end user that's a very nice tool to to play and to better understand what ot is let's move on now to crdt so crtt is more like a distributed data structure you may have contours you may have released that sequence maps and so on the idea there is that uh message flowing from from clients to to all other clients so the processing of that message must be commutative associative and independent so communicative allows you an associative allows you to to play with time so if there is some let's say some network breakage and so on whatever the order the message i receive is fine and in the button if you receive multiple times the same message the result will not be affected you have also to deal with crdt with storm storms because when you delete a message actually this is not deleted in the history it's just marked as tomstom and probably if you want to ensure more performance crdt you need to garbage call those storm storms you have different well two main modes the state base and the operation base operation just sends the changes and so it's maybe more performance in terms of network but needs more logic on the client to merge those changes and it is not only for real-time collaboration so document editing it's also used in well-known distributed databases like redis react or cassandra it is also used for content distribution the idea there is that you have a revision log with a list of changes when you want to make a change you append to that revision log your change and the end user must replace the complete like history to get the result so once again i invite you to go to websites read a lot about that there are some nice blog posts for uh shown here uh in in this slide where it takes you step by step with user one user two typing something and all the conflict resolution is is unsure there are different methods to resolve that conflict here the the blog post i linked to use is the id of the user just like a way to to add these changes before or after if there is some conflict nice visualization again very cool to understand what's going on uh text dash crd compare dot search dot sh it shows you all the data structure evolve when you type something in the editor for different libraries and on the right hand side of this slide i show also a lumino example uh so lumino is a library we have developed inside jupiter lab and when you type o for example you see inserted text or this is the kind of message you see flowing the network cool now time to move to the jupiter specific requirements so you all know what a jupiter notebook is it's a list of cells with output areas and you may have many notebooks open in jupiter or specifically jupiter lab because this is our first target jupiter app so many notebooks many cells in a notebook output area per cell and all that models the data models must flow across different clients we need to ensure for example that when a cell is moved up or down or when a cell is inserted that all users see that action in their notebook we must also ensure that when you run some code that the output is updated in the other notebooks user and this drive to interesting questions because what if a user runs a code which overrides my my variable so if i have a variable x and someone typed x equal something my variable will be erased so we need in a way to introduce or to manage shareholder kernels that works for gpt lab has begun last year in 2019 and it has been done by uh some uh core jupiter lab contributor and it has resulted in what we call the lumino data store lumino is a widget toolkit which backs the jpet lab solution it's a separated repository and in that separated repository there is a new package introduced which is called lumino datastore which managed that model and all the the message are received send propagated and so on that work as let's say completed in with a poc a proof of concept end of 2019 and at that time there are a few issues or questions that has been raised and quite difficult to to move forward and this is why we have decided as the gp lab community has decided to move that initiative from gpt lab to make it a dedicated jupiter rtc project not only for jupiter lab but for any jupiter front-end for any jupiter component application which needs a real-time collaboration so if you clone today the gp2 labs rtc repository you will get that a few components uh you can read more on the reader docs we have uh b weekly zoom calls to to connect with users to discuss together what we want to do and also we are funded by czi grant and which is very good because we at least we have quite some manpower to to deliver and to let this project evolve we like to be specified we like specifications we like uh having a good model on the server side and if you want to to read more of that data model you can go once again to the website uh this is an extract of that website you can see there that uh we want to to move from the client specs to the server and we want to to have all the notions of jupiter as today which is execution code kernels kernel status and so on good now you have cloned the repository what you see is a set of packages we have the what we say with the core packages and you can see there are the nodes the super nodes the relay the jupiter jupiter is the one with the data model and the relay supernode is more like the server aspect which hosts the the the receive and the send of the messages and the node is more a client which goes with react.js to his software develop application development we have some tools uh debugger a dummy store which is like an in-memory uh server and we have also example to showcase what we do because we want to i mean the user wants to see something it's not only having a server or having some libraries but he wants to know how to use them and we have the lumino first example so this is the one at the right top of this slide and this is i mean plain let me know it doesn't use the other core library but it's just something we want to to show uh just to have a basic knowledge and understanding of of what lumino has to offer today we have then two other examples which does use the core packages so the first one is a to do application while we all know all know what to do app are you you type to do at least to do to do today i have to do something and uh that to do action is also shown uh uh for to the other user and you can delete also when your action is done and talking about more specific jupiter we have one jupiter example which allows you to see the data model uh with the debugger so on the right hand side of this slide at the bottom you can see there the debugger where you can see the different tables on the server we have also updated the jupyter lab integration so today uh you can use with one of the latest master jupiter lab you can use uh rtc uh this works fine you can move cell up down you can uh um select some text and the selection will also be shown in the other notebook so here on the picture you have uh one browser and a second browser firefox and uh and uh and they all uh pass the information without any issue it works fine we have an updated uh version of the work which has been done in 2019 but we need to let this work evolve to use the core packages we develop this one is about graphql so one way to move them the data model from the client to the server is to use graphql which is uh which offer nice uh interaction between a client and a server and which allows you to persist that data model on the server in many different uh implementations and what we want to do is move from uh that graphql interaction and build collaborative features on top of the graphql channel also that uh collaborative uh way we have that lumino library the jupiter la premium uh but this is not the only one we want to address uh we also want to leverage other existing real-time libraries like ygs like auto merge like freed and so on we we are looking to build like a layer which allows the application via our packages to to use existing rtc libraries and to serve a lot of user we need to be performant i mean you've if you remember we you've seen this is the payload of the message just for one character uh which is not uh tiny uh and so we we need to make sure that the different operations we we serve our performance that the the payload of the message are not too heavy and for that we we are looking at existing benchmark and we had to to add our implementation to those benchmarks we also need a scalable platform if we have a few user a simple server we will be no issue at all a single and simple python or node process will not be an issue but if you really want to scale to to a lot a lot of user uh you need to deploy your your service uh on something like kubernetes which is a distributed platform management and also maybe you you want to transfer messages in a more async way uh across those distributed services and something like kafka which is a distributed message queuing system is also something we are looking for yeah so thank you for uh being here with me i hope you you had good overview is just an overview of what we do an overview of the algorithms we we use and if you want to jump into to the code with us i invite you to try the examples this is a very first step you you could do so just go to jupiter lab rtc on github and go from there to the documentation on the wizard docs and follow the uh the the very simple steps to run those examples we are also looking for user stories because well we need to hear from you we need to to know what you need so come and john open an issue come to zoom b weekly meeting and if you are more looking for helping us on documentation or design we we need you we need a better docs we need a logo and so on and if you just want to use the library that's fine also but be be aware that this is still a work in progress so the api is quite unstable i would say so once again thank you to all thank you to sis to see that eye for the grant and have a good time thank you