How I read and annotate ML papers

Key Takeaways

hi everyone how's everyone doing all right so first thing i'm going to do because i'm still very very new to this i just want to do a quick check just to see if you can i'm here here and see me over the other side just let me know just me know on the chat i'm looking at my chat here just give me an indication that you can hear and you can see me i'm so does check in here can you all hear and see me [Applause] oh perfect all right that's good to hear so a lot of you coming in just now welcome all right so i'm just gonna wait a couple of a couple of seconds here a minute or two just to give people a chance to get into the stream i'm quite excited for today i've been promising to do this sort of uh i i don't know if it's a meet up or whether it's a talk or whether it's you know an explanation of something i don't know what it is i feel like it's just a very informal chat and yeah i'm gonna i'm gonna share a few things that i learned over the years about how to keep myself really focused on doing ml research and you know some tricks i learned along the way some strategies that i use every day hey you all can you all see me i know some of you must have left already sorry about that um just let me know if you can hear me on the there you are can you answer me i know some of you must have left alright yeah okay great i think this can still happen then in that case so the thing is i was trying to you know try to set up something here and then um kind of a new setup i tested it you know you know how things go when they go live they break so i'm trying to now use zoom for this one trying to use another software because i really wanted to show you a demo of you know how i take notes and what kind of software i use and what kind of devices i use as well so i unfortunately won't be able to show you all of that but i can still show you a pretty cool demo still on my desktop and hopefully that's that's reasonable and it's pretty close to what i usually do but i have a couple of slides that i want to share with you so let me just bring those up and then i'll share my screen for now so let's bring this here yeah this is this is completely new to me so bear with me okay i'm trying to get used to all this uh you know with screens all over the place so it's it's a really really new experience for me but i'm trying my best here okay so i should be able to share my screen now actually in fact let me just bring this up here and then we should get started with the talk it's not a it's not a really long talk anyways but um you know it's it's just going to take a couple of minutes just to briefly talk about what i wanted to talk about today while still still giving me a little bit of issues but oh man okay it's so weird because i do have i do have a nice setup that i use but oh what is going on now just give me a second yeah oh man something is going on no it's not i'm not sure what's going on okay so okay so i'm trying to figure out because my computer is just really really um behaving crazy today but i'm trying to figure out something here because my screens you know my browser is not functioning everything is just not functioning so i cannot move things around how i want to move them so everything is just chaotic for now sorry about that um oh wow just a complete mess oh what is going on with this one why is it not allowing me to close okay all right i think i figure it out okay so are you all there still yes no okay good thanks for sticking by let me just try to share my screen now and then we should get started okay sorry about all of those things um it's just it's just this this this computer today is just not really coping with me and it's just messing everything up for me so let me share my screen now okay and we should be good to go okay so now you should be seeing my screen right so i think you're you're able to see onenote is that is that okay can you see one note on my screen yup okay thanks thanks it really helps when you send me back a response he lets me know where i am okay so let's get started so what i wanted to talk about today as i was saying i want to talk a little bit about you know um how i what's my my thought process when i try to inform myself about things related to machine learning specifically the research side um obviously i'm also part of the industry so i do i do a little bit of also work on applications but mostly i'm focused on on the research side of things so i always try to keep myself updated and everything happening and i have different ways and different tools that i use to to keep myself informed of the latest machine learning research and you know that use all of that of course uh to help people in the community to write blogs um you know to write newsletters and and i do all this educational stuff which is really helpful for people uh but behind that you know i need to do a lot of work and and inform myself about what's going on and that's a really really tedious task so over the years i've tried to basically come up with different ways and how to improve my way of reading papers and taking down notes and doing all the stuff that regular researchers would do and i have a set of tools that i also use so anything that i discuss here by the way you know it's not a guarantee that it will work for you obviously everyone has different ways and preferences and they work with different tools not everything is going to work for you that you see here but these are just some things that have worked for me and and i've kind of tried to optimize you know my my way of doing research and taking down notes and doing all that stuff because obviously there's a lot going on and especially with the community so i really need to be good at this so the agenda for today will be tools for researchers basically discuss a few tools that i use and hardware stuff because people were asking me before okay what kind of setup do you have what kind of hardware do you use and these sort of things and also resources for staying informed within with ml and nlp paper so what kind of resources i use what kind of websites i use and so forth okay and what i'm paying attention to as well when i read a paper so what are my learning strategies uh you know how i read a paper what's the structure look like in terms of learning and how do i pick topics right those topics have to be relevant and what do i pay attention to so before i even read anything before i grab a book before i go to a panel discussion before i do anything i tend to always i always tend to you know ask myself the question what do i need to pay attention to so there's a suggestion here please can you collapse and leave only the notebook with the slide and zoom the notebook view so let me just try to do something here then i think that's what you're able to see right i'm not sure what you were referring to there anyway so i'll keep moving for as long as you can see the slides that i'm showing here i'm using onenote by the way so as you can as long as you can see the slice is totally okay then i i wanted to do a demo a live demo but unfortunately i won't be able to do it uh using the device that i use but i'll try to use the onenote that i have here on my desktop uh which is not the most optimal way but i still think it's gonna be helpful to show a few things and then there's going to be a discussion q a you can ask your questions while i go along as well just feel free to ask questions and i think a lot of people are here to help as well so if you have anything that you want to ask i know a couple of folks here are you know interested in research as well and they use their own setups and their own tools so i think it could be a great way to kind of share as well oh okay so let me just i can just zoom in yeah i've never done a presentation this way i just wanted to just use onenote because otherwise i'll be just dealing with a lot of screens and right now it's just this computer is not really the most optimal i don't know for what reason all right so then have a discussion i think that would be great as well um so what are the tools i use for doing research uh so i i tend to use these tools here that you see um as you can see it's very it's just a very short list i'm not focusing too much on on you know buying expensive hard well it's expensive hardware in some way but um you know i won't buy something too fancy to say and and and what i'm mostly interested is something that's very easy to use um it's going to help me with focus and also something that's portable because i move around a lot i do travel a lot my actually my profession at the moment which i didn't speak of and that's not important here but um a lot well i'm always traveling all over the place so i always needed something that's portable i think for now i think that's not really that important but i think as you as you as you know try to move around a lot and you go back to your offices and support i think it's really good to have a device that you can always carry with you you know spend some time doing the the research also on the road it's a lot of fun doing that i'm sorry i said earlier also with the with the preferences um you know hardware software you use all your own softwares i use these ones mentally for the uh for for jutting down the well for recording the like the bib text that's mostly what i use it for and i use it for doing some organization of papers and the onenote is what you're seeing this one i just use it to basically annotate my papers and that's what i used to read i do combine it a little bit with the with adobe reader i like to use adobe reader because obviously it's um you have the actual pdf the disadvantage as you will see in a bit with one note is that when you actually render a pdf on onenote you can only you know get access to a printout which is just images you don't get access to the raw pdf which means you cannot really do searches over that that's it's a it's kind of a drawback but i work with that and i do combine it with something like adobe and it does help me out a lot and so i also use the so i have an ipad which i bought like a couple of months back and you know i was not really an apple kind of guy i was just i'm just like focused on really really like traditional you know software and hardware as well not focusing on too much and fanciness but i realized that i really need a device that's very very portable and something that goes along with me easily and so i decided to make that investment to buy an ipad mini and i'm not like trying to advocate for apple any at all here because i'm not really an apple person but i think it's a great device um it does its job really well which is it keeps me focused on what i want to do right i don't have many apps installed in it i just have you know the regular apps and the apps for reading and for reading my email and that's about it you know if i want to have something like uh like entertainment something like youtube or something like twitter that's that's you know for another device like my phone but this device is just mainly for reading and it really really really helps me to focus which is kind of very important here okay just moving along um now these are the divide the resources i used to basically uh stay informed with ml research so these ones have found pretty useful and you know i keep the list very short right like i said one thing i want to do is just keep myself focused and right i kind of go through these ones every day so i would go through a couple of um maybe a couple of pages i won't go too deep like for instance for archive or go maybe last week or something like that but i won't go too deep on that and i'm looking around for four interesting papers uh particularly for nlp and machine learning and those topics that i'm interested in and i use archive sanity archive sanity is just this website i don't know lately has been kind of giving issues um it's not working sometimes it's working sometimes i don't know what's going on there and who's maintaining it but it's a really really nice website it gives you a good feel for what people are looking at what people are reading what people are sharing on social networks and these sort of things right so even though it has this kind of little bias there i think is pretty helpful and i would want to add reddit to that as well although reddit is kind of nuts not a really reliable source in my opinion so but but it does help sometimes to find something really interesting and twitter has helped me a lot so twitter is where i'm mostly active and try to look for interesting papers at least that you know maybe past colleagues maybe people that i'm i'm following and and they're doing a really interesting research that i want to know about so that's how i kind of stay updated with those works um paper sheet code i think the one that i really strongly recommend so papers and code and semantics color i really love these two a paper sheet code is if you really want to stay up to date with state of the art which is you know what that website is about it has a lot of papers new papers are coming in and it has code which is great if you want to test out something and it's all about reproducibility the code is there you can you know give it always a run so if you don't know about that check it out be really really useful as well in terms of resources semantics color i've started to play like a couple of weeks back started to play with it and i really like it because it has this kind of like a recommendation engine which i didn't see other websites have and so you know i use a combination of these so the cement the semantic actually it uses some kind of semantic search i guess to recommend some recommend some paper some related papers to things that i'm following so i create my own feeds and it's creating all of this stuff um always recommending every day recommending papers for me that are kind of relevant so it's pretty useful i really like it i i like where they're going with it and i like the recommendation engine um i have it connected with my inbox so it's always sends me really interesting papers that i use to share into on twitter and support so it's really helpful um in terms of yeah that'll be it an archive of course i to go to archive although that's kind of really tedious obviously because you go through each category and you will see that you have um a ton of categories right so that was just spend a few a few like minutes on each one of the categories i would go for like language for instance or go to computer vision and then i would just try to keep myself focused on those ones that i'm interested in but you're going through the list manually it's also really helpful although probably it's not the most optimal way but i still like doing it because that is what i used to do before anyway so but things are getting better i think the community are building resources uh for us to to to inform ourselves about machine learning and what's trending and so forth so i think there's a lot of tools out there that are really helpful these days okay so to the middle of the talk i guess the learning strategies for paper reading so what i'm going to use here oh made with ml so someone made a comment made with ml on the chat yeah i think medium with ml is great so you see a lot of papers right you see a lot of um notebooks you see all of these different things that are nicely created by people in the community which is really great right that creation really comes in handy and i think they found a really nice like uh formula for scaling it right i think the scaling part um you know they're doing well on that end and i do use it sometimes as well uh to find a couple of tools and so forth and and also just to discover things that people are sharing so it's pretty helpful that's what i i guess yeah so the learning strategies for paper reading what i want to maximize for is just a deeper understanding when i read a paper i want to you know before reading it obviously i need to make sure that it's something that's useful for me and one way i do that quickly was to read the title and then go through the abstract i think everyone does this right but i found over time that you know abstracts are not the most reliable way of really figuring out whether a paper is interesting or not um you know usually what i would do is just just make the little extra jump and go into the introduction and try to see you know how this was a narrative around the paper and that usually is really quick for me because i know what i'm looking for and this is this goes back to what i was saying earlier what am i paying attention to um typically if i want to for instance it's a new method or something like that i would you know definitely look for those cues in the introduction and if i don't see then you know if it's poorly written i know right away and i can really sense it right away if it's poorly written you know if i'm looking just for contributions something like that and that's not in the first page or two then it's obviously not gonna be a good read for me um results maybe it's about results and those things need to be really summarized in the first page or two and this is one way we quickly tell whether a paper is really worth my time obviously there are different ways you can do this but this is what i use and it all depends obviously on what you're looking for if you're looking for more like you know architecture wise or maybe architecture aspects of machine learning maybe this is a different you need a different strategy to figure out what paper works what paper is worth your time so one thing i do have emphasized a lot over the over the years is what how and why i learned this in my phd the hard way i when i did research these are the three questions that i used to center myself around the things that i was working on so what the hell and why what is this about so what is this paper about right what is the main contribution of this paper you know how are they how are they addressing this right what kind of techniques are they using those are the things that really help me to pay attention to um to a paper uh whatever i'm looking at whatever topic and then why why was this proposed i need to see that like in the first paper in the first page or two and typically typically good writers and good papers usually have these three things all covered in the abstract and introduction you will get a feel for it and when it's once it's there then you're good to go you can go into it to the next page and so forth this is the the method i use for quality checking um obviously again it all depends on on on on the field itself but i think in machine learning this formula has worked really well for me um over the years so i try to optimize for that and i always maximize myself for deeper understanding for whatever whatever i'm reading um so what what did i learn right so this is like kind of my anchor point so um what am i learning here if i read through the first two pages and i'm not learning anything you know i said it's time to quit this paper because there's so many other papers that i want to read or maybe have or maybe have some some something interesting that i could read and i'm just wasting my time so i always want to know you know what am i learning here and what's my anchor point so one thing i want to say here is uh so this is the time that you spend this is the the point where you're going to spend some time taking notes right so you already started to think about taking notes and taking notes is i don't know if how you read papers but when i read a paper even if it's a paper that i'm going to abandon at some point i always try to write some notes and i write some questions along the way about what i'm learning and about how this paper is is useful for me up to this point whatever point i was if it was if i was at the introduction level or whatever but i'm always writing notes even if i abandoned it i will always have some notes there and i use that just to basically help me with recall um obviously it all depends on how you read papers and how you take notes and what software you're using but i think onenote has helped me a lot with this tremendously and that's why i've never quit onenote because obviously when i'm writing right i just write whatever i want to write the point i want to write whatever question i'm going to come back to and i'm keep annotating this along the way and i always have this for recall and i always go back to that paper at some point um if it was a paper that i was that i read through um all the way if it was abandoned then i just categorized that somewhere else so so what did i learn right so when you learn something right you want to take notes that helps with your call as i said but also feel i think one recommendation i would give here is to always try to write or try to put it down somewhere right even if it's not on a onenote page here but even if it's just on your text editor or something like that that really helps a lot because it really helps you to basically test yourself if you're understanding what you're reading if you're not understanding what you're reading then i think you're just wasting your time and like you argue against that that you're wasting your time if you don't really understand if you don't really write it and and try to go over it and try to understand it this way um so i always try to you know write stuff so that you can recall it or later on um the next point i want to make is about um so the one on the right are just reading strategies so this one is more like a top tone so what is this about the top tone and these are just reading strategies right so here i'm focusing on maybe so do i read the paper from you know introduction all the way to results to i don't know the the conclusion or do i go the other way around or where do i start i think this all depends on what you're looking for and this is what i was saying what are you paying attention to so for instance if i'm if i'm mostly interested to look at the results right maybe i know this sort of methods or family of methods maybe you just want to look at the results and i would just go straight to the table and try to look at the results sometimes i don't even know need to go to the to the paper because websites like paper we code already have those results for me in a table so i could just quickly tell so this is the reason why i think the community is doing really great with those tools because they already help you to get an idea of what this paper is about and that really really helps you to maximize and to make good use of your time and so if it's results to methodology right so if it's you're going to look at the results they are interesting now okay you're curious about you know what was tweet right what uh what happened in this architecture what was the design choice all the set of stuff hyper parameters whatever okay so that's what you get with them papers but you could do it the other way around too so let's say you're interested in you know methodology right so you want to know what the architecture is about you're curious about it and then you go directly to that as opposed to going to the results so it all depends so i think the the results the methodology um i think it's more like a top-down approach and the other way will be like you know just going trying to go sorry so this one would be more from if you want to go from bottom up or something like that right so if you want to go bottom up uh i think you'll maybe want to focus on first introduction the motivation behind this work then you go into architecture and so forth so that's that kind of your bottom up and this is great for people that are getting started and people that are learning it's always good to always start with the bottom up makes a lot of sense but for more experienced readers i guess the top tone is something that usually the strategy that you would typically use um so i highlight some along the way i want to highlight top five relevant papers and this is how i collected my um if you're you know what if you are want to write a paper at some point based on the the research and the literature that you're reviewing you may want to take down notes of these relevant papers because these are really easy to lose especially if you're reading a lot of papers so i think annotation tools like onenote really helped me with this you know i use a different color and i kind of label that and you know i won't label all the papers it's just maybe a paper that i thought was interesting and i would maybe aim to highlight at least five relevant papers obviously this has to be a really good paper and it has to go it has to have really good references i always go to kind of a skeptic approach what does this mean i don't like to use the word skeptic too loosely but this is something that i've learned in in the field that i want to you know look at first of all when i go to read a paper i'm now a skeptic you know about the results i'm a skeptic about the architecture the design choices and so forth this is what this is with the mentality i always take it's not about being negative to be honest it's more like i think it helps me to kind of open my mind and being very open about what this work is about and you know trying to look for faults if they are and if they are false it's just going to help me to figure out whether it makes sense to keep reading this material and so forth but that has helped me a lot and and how do you take a skeptic approach well uh you know answer like ask the questions right ask the tough questions what is this question everything that you see everything that's confusing you question it maybe it's not even maybe it's not that you don't know it maybe it's that you know the author didn't really communicate it well and this do tends to happen a lot with machine learning um papers i'm telling you it happens it happens a lot okay so i always take that approach um so yeah this one is more like i label what i don't know it's like i want to expand my knowledge base so i would label it and i go back at it i i think i mentioned this before and then i also label what i know what does that mean so and how do i use this again this goes back to the other point about recall so i want to label those things that i already know as well and how very important here how does this connect with what i don't know because you're trying to create the knowledge base for yourself right somehow you're trying to connect things and if things are not connecting you're not going to have a good experience so you know if something is not connecting maybe it's what you don't know right so you label that and you label the things that you don't know and then you try later on to try to trace it and try to see if there is something there that's missing or there's some some gap here and what why is there a gap and so forth so this is this is a good approach i think um to make the most use out of your reading time and try to understand a machine learning paper i think these are things that could generalize pretty easily to other domains as well but you know i'm talking about the context of machine learning here obviously so any questions so far any questions from iran um ask away okay this is the time to ask away anything any questions okay so um one question here from money so how do you how do you how are you able to connect with what you already know yeah so okay so great question right so this is this is related to the the second the second points there um so i think when you're kind of you know reading reading a paper right um i i think you already know some stuff yeah you should know some stuff otherwise um reading a paper probably is not not the the best place to start maybe it's even reading a book or even reading a survey paper or something like that right so i'm assuming here you already have some some knowledge base that you can ground yourself and whatever you're learning um so you will always you know try to annotate the stuff that you already know so those stuff will probably be highlighted when you're annotating your paper right so you would have those like maybe you know in a color and a specific color uh maybe some phrases or something like that that that resonates with you or something that that you know um and you don't have to do it by right by the way but i'm saying it's really helpful when you want to connect something with something else because what i've learned is that when i do that kind of annotation um and i go back and try to read the paper again because obviously i go through it a couple of times um at some point that point was there and i made that connection that connection is not so obvious that's the reason why i make the point here but it starts to become more obvious as you as you as you go over the paper again because you already have those highlighted and you know you already have that in your mind that at some point maybe this is somehow connected to something else because i've actually annotated and it was important for me somehow um so you're kind of taking a gamble here but um what i'm saying is that those methods are really good for recall anyway so even if you if you cannot really connect the dots with something that you don't understand in the paper it's there for recall so this is kind of part of the the recall part that i was talking about okay so really good questions here um let's see wow so so so much good question so i've seen again all of these things right take them with a grain of salt these are not things that are gonna generalize or or you know are gonna work for everyone this is what has worked for me i'm just sharing my experience from my point of view so maybe some of these things could be um could not work for you but you know just take care with a grain of salt and that's what i'm trying to say okay the other question is i've seen that use different colors when underlining the papers is there any rule of thumb you use for the color so i don't know if you have noticed but a lot of people are starting to kind of put out this annotated papers and these sort of things in our community and that's great i think because it shows that people are actually using these strategies right this kind of different colors and and all these different drawings there to help guide whatever you are reading so it actually is really really effective and some people of course are more like on the visual side of things like me i'm always trying to draw or make a like scribble something um and this has obviously this is connected with the colors you use and the colors it's all up to you how you want to like how you want to choose it but i do tend to use like the the yellow color is something that um maybe is of interest for me or maybe it's something that when i go over back again through this paper um those are the things that i probably will take a look at and so one strategy that i do use um i try not to highlight to use too much of the of the of the yellow although sometimes i would use a lot of the yellow um so i would always have like this thing where it has to be a little bit spread out if it's you know i was continuously doing this in a paragraph um this yellow yellow kind of um i'm highlighting i think it's just it's just a distraction at this point so i always try to strategize don't don't try to abuse it too much when you're annotating the paper and that's how i would generally use it so the colors again is just what you prefer but yellow is something that i use to kind of you know calm my attention that's why i use the yellow here it's just to highlight something that's um you know that's that i thought was really really useful or something new there that that really resonated with me um but yeah there are other ways you can so i would i would maybe use so i like to use the red obviously red is more like for danger right i don't want to talk about that philosophical part of thing but i think the red is really great because it tells you okay you know warning something's happening here maybe i'm not understanding something i would use that um so fortunately i don't have an example that's what i wanted to show but um red is a really good color to to um to point out something maybe that you didn't understand or maybe a huge question mark there so when you're preparing a paper how do you prepare the state of art i didn't understand the question to be honest maybe you can rephrase that question go bernie maybe that will help me a little bit yes iterating and coming back again helps i strongly agree with this point i think spending all that effort and that work trying to figure out what paper is useful for you and what's not all that you know filters that you apply really makes sense because obviously if you if a paper really does make it through your filter um you know i think it's worth reading again right there's a reason why you actually chose the paper maybe the first time it wasn't so obvious but it doesn't mean that it's not helpful it means maybe that you really didn't understood something and that's the reason why i think it's really good to label whether you know you know how many question marks you're going to have there it's going to really determine whether it's actually the reason why the paper is not good or maybe because you didn't understand something so that's why i think annotation is pretty handy because of that because you can ask questions and you can tell yourself whether um whether it was you whether it was the paper or whether there's something that wasn't explained so well most of the male papers have mathematical equations how do you help yourself to understand those math oh watch this is a really really really great question so when we have our paper discussions i think people come mostly to try to understand the math most people actually understand papers that we try not to choose um to to to um papers with too much math uh because i think it scares people a lot um but it goes back to that question that actually that point that i made earlier right so the samsung approach um if you are basically a beginner um of reading email papers and maybe there's you don't have uh like a strong background like i would always suggest trying to use the bottom up approach um if you're you know jump into results jump to methodology ahead of time without even jumping without even going through each one of the steps um of the paper this this will lose you entirely so in order to understand math what i would typically do so this is the reason why i love when the community kind of plays these these rules and these policies to actually put code along with papers um and that's one of my filters actually if the paper doesn't have code it's more likely i'm not going to read it unless it's a paper like an opinion piece or something like that um it has to have code that's one of my filters because if i do if i do um and most good papers have code today which is really wonderful and i'm really glad the community has pushed for that most conferences have pushed for that and that's really helpful because if you don't understand something maybe some math wasn't clear um you know the the the code will will be your you know single source of truth your code will explain to you what's going on there maybe there's some kind of variant of a loss function maybe there is something um you know that was done maybe some normalization on the data something was done with that was um you know represented mathematically and maybe it the code can help you associate that i think there's in my opinion there's no really easy way to um unless you really have like a heavy met background there's no really easy way or shortcut i believe to to to understand what the math is saying so i think the code is one one of the ways that you would desperate this um yeah hopefully that's useful for undergrad and ml newbies how do you recommend to read papers if we're entirely new in the field i've tried to read some papers where i have no clue about like like a half of what it's written i guess i guess right maria i guess um i guess that's the reason why you came here um i don't have a straightforward answer to this one um what i would say is uh like i would recommend if you're a beginner i would strongly recommend you to always start with survey papers there's a reason why survey papers are there um obviously survey papers are more like you know high level they don't give you too many details in fact they don't even they don't even have math right they don't have any math they don't explain um they don't explain the the design choices for some architecture or anything like that they don't really go that deep but i think getting a sense you know for what matters in the field i think this is a really good way to do it and i think i remember really well when i started as a researcher when i started with with my master's that's when i started research because in my undergrad i didn't really do that much research i wasn't interested but one thing that really helped me out was books i read a lot of books i think books are really great they are kind of undervalued for some reason but there's a lot of books for instance books about text mining books about uh you know data mining all of these concepts by the way that's how we used to call it before some professors could tell you about that like older professors could tell that it was data mining then it became data science and it's a lot of names it has transformed over the years but anyways going back to that point i was making uh i think it's always good to check what the history was at least for what you were interested in so if for instance in my case i was obviously interested in nlp i was so fortunate because i got recommended from my advisors and my professors to go to those books so there's a couple of books that were recommended and i was always go through those books and try to do exercises and it really created a really nice foundation for me before i even started to read papers of course before i even started exploring that really helped me out because obviously you know i know what i know what mattered at the end of the day when i when i went through the book and maybe um even the book has some kind of references as well that i used and if you notice if you go through those books you will see that those books actually refer to a lot of survey papers and this is how i ended up with survey papers as a next step so survey papers i couldn't emphasize more if you if you really want to know about survey papers i do share a lot of survey papers although i don't think i share enough but if you use things like semantics color um they have really good recommendation agent you know all you do is just type survey and and it will create this kind of feed for you where you will see a lot of survey papers and new survey papers all survey papers and journals have this a lot right obviously journals um if you're only emphasizing on conferences conferences do not have this type of papers it's mostly journals and obviously archive wow so many questions uh i'll try to let me see i have maybe 10 more minutes i want to do like a quick demo of how i annotate although this will be pretty a pretty lame demo i will feel bad after this after doing this demo but i'll try it and this on my desktop i'll probably have to use i have to use my my cursor here and everything but uh let's see if this works but i want to show you i'll take five minutes of the time to show you how i go about reading the paper just to give you a demo let me take one or two questions here more and then i'll do that so one good question here from mark do you follow summarization type of media such as yannick culture youtube channel and if so do you have a few re recommend recommendation right recommendations um so so this is this is a great question because obviously there's so much different options right so you have papers you have like i said those websites that summarize statistics for you about papers which is really great has helped me out a lot um and there you have media right so you have like youtube videos then you even have like shorter videos you have long form videos you have all these different um kind of things i think it all depends on what you you know what what you prefer as a medium to learn i think most people i guess like visual explanations they're like you know does it that explanation really makes sense to them um as opposed to reading a paper i think those are really great by the way um i'm not too familiar with the i know yannick i've seen one or two videos but i'm not sure about you know the entirety of what he produces but i guess uh having that as a maybe as a way to kind of get a like a quick overview so to speak of of what of what the paper is about i think it's it's pretty reasonable to to do that and let's use what we have those resources are there for a reason and people are spending you know their valuable time to do it um and you always try to see for quality right if there's something that maybe if someone is claiming that they want to explain something to you and you didn't understand it after that explanation then you know those are supposed to be red flags if you understand it maybe you didn't understand something maybe because um you know you missed something or maybe you need to create more background knowledge for you or something like that um maybe uh you need to maybe focus on other other kinds of mediums maybe survey papers maybe other stuff that i was mentioning yeah but there's no straightforward answer again but i think you know having videos is really great um even myself i want to do videos as well to help other people right so there's a question here from bhavani hopefully i say the name right i'm sorry if i didn't get your name right i'm poor with names i should be better with names because i'm actually a trainer an educator by profession but i struggle sorry about that so when you read a paper and it refers to a bunch of other papers should you read all of them to get context for example attention before transformers this is such a great question so many great questions here i think i'm going to take a log of this and maybe try to expound or maybe even refer to these questions at some point maybe through our blog or my twitter feed but this one is great because this one is something that i'm trying to build a solution for and i think it's really helpful to give people some kind of guidance and easier guidance and this is goes back to the medium part that i was saying maybe people don't want to read a survey paper because it's a very lengthy paper maybe uh maybe all they want is just like a like a short list right items of items of papers maybe that they want to go through that really helps them to understand a particular maybe architecture or something like that like in this case transformer so i think websites like papers record they actually have like a like a new feature it's called i think it's called methods and that's what they're aiming to do so they want to give you this like uh history of different architectures and what it was used for i think it's so helpful right it helps it tells you like what task is used for and how it has evolved and what new methods came along the way uh and how we ended up with transformers um obviously there's some improvement to be made there because you can actually improve the storyline and give people more information there but i think that's a really good first step and those resources do help uh to create a better medium for you to to to go through um you know to understand what came before what okay so one jack is asking jack is actually putting me on the spot so can you show how you transition from paper to understanding code with a live example um so that's what i want to do later on this is why i wanted to test the stream today it's kind of like a test as well um because that's something that i want to try so i want to try different techniques and i think i promised uh our paper discussion uh group that i wanted to do more live coding because i think it makes sense i mean i think for people you know they're afraid of getting into the code that's that's totally understandable if you're starting um but i think there are ways you can actually do this and you don't want to start with a really hard example a really hard paper you want to start with something that's simple right you don't want to start with the transformers for instance because you're going to get last completely and that if you're not aware of these techniques like you know the history of it and how it came how transformer came to be you're going to last pretty quickly so try to start with a very basic method i think um trying to do something like work vect which is a very simple paper and there's so many code examples out there and try to understand a few concepts you know um that i think that helps a lot i think that's that's how you would go about and this usually does help and that's what i usually recommend yeah so i'm going to try to do more of these but i'm going to basically try to do basic examples and move on to more advanced examples what is your reading strategy for writing the literature review of your articles is more about writing it's not more about it's not about reading i think the question is more about writing um so there this could be another talk maybe or something like that about writing because um i love i love trying to follow best practices for writing that's something that i always try to find good writers online and i follow them and i find all the strategies and i've used that over the years um you know to improve my writing and i think in the last hack i i suggested a few as few a few people that i that i do follow for that and you know writing is so important right communication communication of of machine learning idea is so important because you can easily easily make a lot of mistakes um i use language that you shouldn't but that i think would be another good idea there wow so many questions here sentence in your culture based on background right i think that's more about writing okay so let me do a quick example here and let's see how this goes i hope okay so i have a paper here which i i hope you can see it can you see it think you can see right so imagine i i want to look at this paper now i'll be very honest i know a little bit about transformers right i know a little bit about language modeling right what am i paying attention to that's the first question i said right and i want to emphasize on those things as a strategy that i use um your strategy may be a little bit different but i think the good first question is what do i want to pay attention to well according to the title this paper is transformer for longer sequences i think i want to read this paper because i think longer sequences is something that i think most architectures that have been proposed in the field of nlp have struggled to deal with longer you know longer sequences such as books and so forth so what am i looking for well if it's longer sequences i want to know why is this an issue why is longer sequence designation first of all right i want to know if the author actually is talking about these points um and the second question would be here is uh complexity time complexity because i have that background already and i know that's an issue right so there's a kind of like a constraint with with um attention mechanisms our attention mechanism has this like linear complexity but i have that background now if you were coming straight to this paper and you were like what is this what is okay transformers relaxing because it sounds cool i mean it sounds a really good name right it's really uh uh kind of like a like a little a big bird i think it's an attractive name um but yeah you'll be a little bit lost right so that's that's that was good to ask why do i come here what's the motivation for coming here um and then the abstract of course it needs to tell me the story that i want to see which is if you look at it i haven't really looked deep into this paper although i've just kind of skimmed it um this paper is separately talking about um complexity it may scare you but you know the author is actually making a point right away that that's something that's important for them and that's important for me as a reader here because that's what i want to focus on in terms of what i'm reading and so i that's that's a good good indicator for me um and obviously i like the example of of virus and pitas such as question answering summarization that really gives me a good indication as well that maybe this author is you know is maybe going to consider some examples or maybe want to explain something um i think those examples will be great and you can see a mention of hardware as well here uh the unfortunate thing is because i'm not using i can see here hardware right this one what i'm looking at uh i want to maybe look for some discussion around i want to know how this applies at least in the context of one of those tasks right that's very important for me to get a good understanding so at least for this paper that's what i want to focus on i want to focus on the complexity i want to focus on what are the what are the benefits in terms of like you know the hardware aspect of it and the question answering and then some application of it these are the three things that i want to focus on and so i go and i try to readjust my anchor points that's what i want to pay attention to and so my approach for this will be more like a top down because obviously i know a little bit about transformers i already said that right so what i would typically do is try to see if there's some results okay so let me try to see by the way if you notice how i use my onenote here is kind of weird because um actually if onenotes is not i have it kind of in the center because i always do i do take notes this side and i take notes this side i don't like to just take on notes on one side i think that's that doesn't work for me i want to do on both sides okay so if i go a little bit down let's see if i can find some sort of related work there's a little bit about architecture but i'm interested in the results i want to see right away what were the tests what are the experiments because that's going to tell me um whether you know i will find what i'm looking for right so let me see let's see like as you you can tell i haven't really looked really closely at this paper but i'm trying to look for that information it's kind of a little bit slow desktop is not the most optimal one so i think we have some some results here i guess this one some are some results yeah some experiments right so this is your experiments right here um let me use a different color for that right so so let's let's see the the results that i'm interested in oh i already see something really interesting here so maybe that i will highlight it with um right so the tfidf um you know what's the point of that but i'm not going to touch on that i'm not going to go deeper on that because that it's not that's not the point of going through all this i want to focus on the results and see what the results are and what kind of um maybe tasks or benchmarks whatever this was tested on so that's the first thing i want to know okay so i go directly into the data set so there are a couple of data sets that were used right so the stores wikipedia blah blah blah but i still don't see anything about tasks and that's a little bit um not good news for me but let's see let's keep going maybe there are some okay now we see it so we have the hotpot qa we have natural iq all of these ones if you look at those transformer base papers the language mowings you always have these as benchmarks that are used this one is like ua task right and these ones are more difficult qa kind of benchmarks as well and then you have the wiki hub which is more about i think it's more efficient answering as well so all of these are related to questionnaires apparently so it looks like okay question answering is great that's what i came here for i want to know a little bit about it and let's look at some results really quickly so you can see here if i'm really analyzing the results i just want to let's see how you know those results compare to um to whatever was proposed before i'm still not interested in these ones yet because i just want to know uh my question was whether it has anything related to those tasks that i was interested in apparently it does so i would read a little bit here and try to understand you know um what's going on what the results how they compare what's the improvement and so forth um and this will give me some kind of indicators whether it makes sense for me to actually go through the theoretical analysis of the paper or the architecture itself because again i took a top-down approach maybe your approach is going to be a little bit different if you're a beginner or maybe for someone that's more experienced those are the things that you would generally do and obviously i understand here this is about question answering there's more like other comparison here with base models you can see here very size models and some reports some accuracy and support and they're using this kind of metric right and it's mostly about qa apparently okay so they have document classification as well um and they probably use something like the cls token from bird but i'm just kind of skimming right just skimming this and i'm making some annotations as i go not too sure what that is so i'm going to maybe that it's not so clear for me just going to put a question mark what does that mean encore decoder task i'm going to come back to that that phrase right there is it's not so obvious to me what it is but i will go back to that i know what an encoder decoder is but i don't know what it means in that context okay so there's a little bit of summarization right that i discussed earlier so the author actually wrote this paper really well that's what i love i love when people do that because um you know they don't mislead you they tell you that they're gonna talk about this and they do talk about it and they present it in the paper and that's that's those are really good good um i think good components of what makes a really good paper you don't mislead um that's about writing right don't mislead and there's even more about genomics here which is which is really surprising and interesting as well uh the genomics part of it maybe there's some conversation here which maybe may be interesting maybe conversations because i know genomics data is it's a little bit more complex than your our usual nlp benchmarks so maybe i could ask something here about you know maybe ask a question here um what's the length of a typical data set related to genomics and you know why why did they explore it here so you can see here dna sequence 32k okay and those are the things right those are the things that i'm combining i look like i keep skimming and i'm on the conclusion already and i guess there's some kind of uh some kind of more explanations uh maybe some even some math towards the end and that's what i would do right so i think it's always important to ask the question right what do you want to pay attention to and i think as an experienced researcher maybe that really helps you to get a feel for whether this paper makes sense or or not and that's what i do um i don't try to use like popularity as a filter i think that's not a good measure um you know maybe it's coming from a really research a popular research lab and i think that's a good measurement for um you know for for deciding whether a paper makes sense for you or it's worth your time but i think asking yourself and be really curious i think that's a good strategy overall and that's i think a good a good um a good approach our argument is a good approach okay so any other questions i mean i couldn't like show you because i have my pen and my other stuff on the side and i i just something went on with it it just wasn't working i really wish i could show you but one thing i would say i just want to make the announcement that i will do more of this um so i will do this very often because i will start this kind of um i'm going to use youtube for for doing live streaming i'll do live stream of paper explanations um you know and i'm going to do it the first few explanations would be more for like beginners you know people that are beginning they want to get into the space i want to make it more beginner friendly and i think it's going to be useful even for people that are you know have a little bit of um of of of maybe some some progress already so i think it's it's still going to be useful and and so so i'm going to build up like this and i'm going to create this kind of content and and eventually i will do more advanced paper like this one and along the way obviously i will share a few things that i that that i think were um important and useful too as well no it's not a butter i always do the zooming out that's why i have it on the that doesn't bother me again it all depends on you maybe this is go it's bothering you right but for me i'm always focused and i want to make sure i use a device that i don't have notifications and i have other distractions um it's just me on the on an open onenote and i'm just writing and i like i said i annotate here and i know here i just interact with this stuff like i think interaction just being doing this kind of interaction with the with the with the paper and writing notes and annotating is just helping me to remember stuff and helping me to to really um you know get what i want from this whatever i'm reading so i think it helps me a lot i love it maybe it's not so it's not a good experience for you but i like it yeah i mean searching papers this sort of things yeah um i i wanna i wanna i wanna do more of that as well i think that would be really useful for this community um you know starting with very basic papers maybe like you know those those things like work to back and then building up so i'm gonna definitely use tools like methods uh from paper sheet code because i think that would be easy for me to follow as a recipe and then you know just build up some knowledge build up maybe record a few videos um doing this sort of um paper explanations and just show you my thought process and how i go about reading this because i do this a lot i do this every day to be honest because i need to do it and then it's part of kind of my job as well and and and i do this because i'm very passionate to help the community and educate the community as well on certain aspects um of the field so i'm very devoted to this yes okay yeah i'm gonna take notes i'm taking notes all of this stuff i'm gonna log i'm gonna write some notes on the side um i'll try to i'll definitely try to um you know write more or do more of these kind of live streams unfortunately today something went wrong at the beginning right apologies for that but i'm experimenting with this i'm pretty new to all of this so thanks for being with me uh yeah so thanks very much everyone i i guess that would be it if you have any other questions just let me know i'll stick by maybe for an extra five minutes because i need to go to another i need to go to another another meeting now okay see you soon you