NLP with Transformers author interview with Lewis Tunstall from Hugging Face

Key Takeaways

the the sort of hugging face ecosystem is roughly split I would say in in a couple of areas so one is of course the open source libraries there are libraries like Transformers which are kind of like the main uh backbone of what we do and they're like there are many ways to contribute ranging from you know squashing bugs to heading and new architecture my name is Sophia young I'm a data scientist and I also organize this book club we read one data science book per month and Louis would you like to tell us a little bit about yourself yeah sure so um I'm an engineer at hacking face I um started off working in the open source team primarily um booking or contributing to Transformers on the topic of like optimization for inference and um I also helped co-develop a course we have on NLP and Transformers which is kind of like a precursor to the book and um more recently um as I was saying at the start of this chat I've kind of Switched gears into the research team to sort of help on the engineering side of things around these topics of reinforcement learning from Human feedback and more generally how do you scale up to like the kind of 50 billion parameter plus range where the sort of you know interesting phenomena seems to appear um before that I was a total nerd doing theoretical physics um and I spent uh several years after my PhD as a researcher studying kind of particle collisions at Large Hadron Collider and it was from that experience I stumbled into machine learning when um a guy in my office was like Hey look at this piece of tensorflow code I wrote it can basically beat you and every other physicist at classifying different types of particles and I was like okay I should probably pay attention to this and um kind of got hooked and I started out like many other people did at the time with Andrew Young's classic course where you you do like machine learning with like octave and you know eventually upgraded to tensorflow one um and yeah now I'm in the the land of pyter which where you know life is a little bit more easy to work with you just answered one of my question is perfect thank you yeah I was wondering what's your transition from physics to computer science hagging phase it's so incredible um so I want to give it a chance for our book club members to introduce themselves if you're shy feel free to introduce yourself in the chat don't have to talk but yeah who wants to go uh nice to meet you Luis my name is Danielle I am a software engineering manager currently uh my team is working uh mostly with data engineering but we also have some data scientists and I mean uh when the guys speak like your book like book of the month it was a blessing for me because uh we we were tackling some kind of issue related to NLP and my knowledge in NLP is like bag of forts and engrams and it was really helpful and I really appreciate like reading your book it gave me so much Clarity So yeah thank you for that now now I know okay we have this Pia problem and we have like okay some kind of ner model that maybe we can use to to do some anonymization and I don't know the the encoding the encodies the the encodes lookup for some kind of nearest neighbor search and then like reducing the document space to do some kind of uh q a it was really great you know you also have no idea how much you guys helped me yeah I'm really grateful that I read your book thanks a lot thank you yeah in fact the sort of story of the book is that um I started out learning Transformers from the pi torch pre-trained bird because at the time I was working as a data scientist and I remember looking at this and going oh I have no idea what's going on and then Leandra my colleague was like hey let's like look into this and then after a while you know there wasn't many opportunities or many things available so there was like Sasha Rush's classic um you know the annotated Transformer which basically derives the Transformer kind of from scratch and then there was like um Jay alimar's really great blog posts about like The Illustrated Transformer and because we were more practitioners we wanted to do something that was helpful to other data scientists and so we said hey let's write a book about this as a as a kind of forcing function to really learn stuff and um my wife was like oh you're always talking about hugging phase and Transformers maybe you should like check with them that they're not writing a book because it will suck great if you do all this work and then you know they release the book so I just sent Tom Wolf an email like out of nowhere and just said hey where these two random data scientists writing a book are you interested in you know co-authoring it with us and he was like yeah it sounds cool and uh that was like you know one of those things in life where you think no way on Earth is Tom going to reply to this random email but I'm quite happy he did and at the time you were not even working in a hugging place no no it was um we were both working for Swiss companies oh wow Paris was Telecom company and Leandra was working for a Swiss insurance company and um yeah it wasn't even planned that we would you know join hugging phase because you know for a long time it was like okay I'm not going to leave Switzerland or something like this and so I think the pandemic kind of enabled this ability for people to work you know across countries and yeah one thing led to another wow and now you're both at hiking face yeah that's incredible yeah yeah we've been there for nearly two years now so that's that's such a great story what's the process of writing this book it's very very technical every chapter is a great read I feel like this book takes me a lot longer than other books we have been reading in the book club um so it must be really a long time for you to write as well I could imagine how long did it take for you to write it yeah so it took us a year I think yeah maybe a year and a few months uh from like the very first you know piece of code we wrote to the final thing um but the the most time consuming part is not writing it's actually making a use case that works so because we were taking this like very practical approach um behind every chapter there's maybe a thousand failed experiments um so in a sense it was like doing data science right like you have an idea or you have a like a problem you're trying to solve and you have to iterate many times to get something that kind of works and that was the the most time consuming thing um and then I would say the the writing itself was roughly split between the authors so we would you know say I'll take this chapter and then send a draft about a month later and then a few weeks later we'd go go to O'Reilly and get it back so the yeah the hard part and this was because we chose to have code we made our lives a thousand times more difficult but hopefully this turns out to be more practical for other people so it's not just you know theoretical yes I really appreciate you have the code there so we can try it out and also GitHub repo very helpful yeah that's right yeah I think except for one of the chapter where you're trained for like a couple weeks and that one we couldn't try yeah yeah yeah unfortunately that one is quite expensive you need uh fifty thousand dollars I think to to train that model but still at least the code is there if you want to adapt it to your application yeah thank you yeah we got a club member introducing himself here in a chat uh hello not sure how to pronounce your name I would not try to embarrass myself um okay we have several questions does everyone anyone wants to start with your questions or should I start okay I can start uh with mine hi I'm Peter um and I have a question actually given all of this stuff that happened in last half a year what would if you were to write another chapter of the book what would it be maybe it's something totally because gpts are not that new maybe but maybe there is something you would like to add like a like another chapter what would it be about yeah that's a great question um we were lucky I think that we released the book before chapter came out um and in fact the four diffusion models came out so I think those were the two big changes in the landscape of what's possible um with like generative models that we didn't uh even touch really in the book and what I would probably do if I was um adding another chapter would be to focus a bit more on like multi-modality so the the idea that like it seems that the future of the field is heading in a direction where combining text with other inputs like images and eventually I guess audio and video will come soon um seems to be where the most kind of progress is and having some examples of it I think would have been quite nice to show um and I think the other element would have been um this uh sort of reinforcement learning from Human feedback uh business that is powering um sort of chat GPT and other models um it's also kind of old like um openai first started exploring this in like 2017 um and 2019 was roughly when they got it to work with NLP and so you know we knew about these things but I think none of us would really have like our eye on the on the thing I think okay that really is going to be the the secret magic or the ingredient that is going to transform the quality of the language models and so probably that would be like another thing I would mention um maybe not yet able showing code and how it works because right now I'm doing these experiments and it's very very expensive um mostly because the the compute and the data collection is itself quite expensive but still showing the actual process and and the the kind of difference in quality that one can get I think that would be a nice addition so along the same line which chapter would you change anything you would uh write differently um so yeah maybe the chapter the let me think so there's two areas I would think about changing um we have this chapter on on pre-training um which is this uh where we train a kind of code generation model and um at the time we we used just like kind of conventional tools so things like um the accelerate library and um Transformers um one of the kind of interesting uh developments in last year has been the kind of realization that there are certain tricks one can use for pre-training to make it more efficient so one of the sort of most impactful changes is something called multi-query attention where in this attention mechanism that we sort of discussed in one of the earlier chapters we talk about how you've got these kind of like three matrices you've got this query Matrix a value Matrix and a key Matrix and this kind of is how you compute these attention scores and um one of the researchers at Google realized you can make this dramatically more efficient if you only just basically keep um one of these query matrices and what this lets you do is it lets you train generated models um and evaluate them in a really really efficient way so the model we train in the book if you want to evaluate it on a benchmark for code models it takes you maybe two days of GPU training which is quite expensive or sorry GPU inference which is quite expensive and with these new techniques around attention you can do it in like 10 minutes so it's uh it's it's a dramatic Improvement in like kind of sort of optimizing your code for training efficiency and inference efficiency that I think would be nice to have included as a way of showing people like you know how you can do things in the real world um at scale um but yeah that's uh you know it didn't exist when we started so can't blame me for that sounds fascinating where can we learn more um I can I can dig up the link actually if you want um oh yes please thank you so it's a paper called Fast Transformer decoding I'll put in the chat is there a coding example hugging face somewhere yeah let me dig it up um so I know there's um so Leandro my co-author he went on to uh form this thing called um big code or the big code project I don't know if people have heard of this um I'll put that in the link as well um and this is essentially a project to try and replicate open ai's codex model so open AI has copilot which is like a product that you may have seen in vs code and GitHub and the kind of original model that they kind of discussed in a paper was called codex and so this big code project is an initiative to try and um essentially replicate codex for python and in this project this is where they um realize that if they introduce this multi-query attention then they're able to um get you know very very fast um uh evaluation um I'm trying to dig up the the part of the code but uh yeah I can't find the exact class but basically if you look inside this big code repo a Megatron you'll you'll find uh find it there okay thank you so much sorry to get like super technical super fast but um yeah no no we have very technical audience here sorry Peter I think I interrupted your conversation earlier do you have other questions yeah I was just seeing things answered before yeah but I have like one more question so given uh how fast everything's happening actually what the skills should we get to be able to contribute to hugging face because as for now for me I know pythorg but it looks like most of the stuff made with pytorch is done and now everybody is using like this jacks or you know stuff still I have to learn this and when I will be ready to contribute something else will be used right so what do what actually to learn to study to be able to contribute to hugging face to their whole open source ecosystem yeah that's a great question um so full disclaimer I don't know Jax either so um I I can't uh I can't make comments one way or the other in favor of it or not um but I guess what I can say is that um in general there's always like let's say Frameworks like for deep learning and there will be another one or many more coming after Jackson and so on and my personal experience has been it's better to know one really well um and really focus on that because generally speaking learning once you know one very well learning the others is like you know a little bit like if you know Spanish you can probably speak Italian a little bit and you know the neighboring languages are not so different um regarding the ecosystem I mean the the sort of hugging face ecosystem is roughly split I would say in in a couple of areas so one is of course the open source libraries and in that respect um there are libraries like Transformers which are kind of like let's say the the launch or the main uh backbone of what we do um and there like there are many ways to contribute ranging from you know squashing bugs um to adding a new architecture so for example a lot of research Labs like from Microsoft and Facebook they typically release like a model and they say hey check out our super cool model but it's usually implemented in their framework like you know Fair scale or Jax for example and um a lot of um the sort of users of Transformers are primarily pytos users and so porting that kind of pre-trained model into Transformers with like a pytorch modeling file that's one kind of usually like high impact contribution especially for models that people are very excited about um and then you know outside of Transformers there are other libraries like for example we have like this datasets Library which integrates with the Hub to enable um loading and processing of data sets we have the accelerate Library which is not really specific to a type of um like framework but more a way of simplifying like distributed training and as with most open source projects it's a little bit up to you like what you want to contribute to and my personal experience has been that when you're trying to solve a problem like in your day job or in something like that that's often where you have the strongest pressure to like make something work and that gives you like the sort of push to contribute back like if you if you want to make I don't know accelerate do something and you're like oh it doesn't do it then typically then you can sort of you know propose it as a feature and if it's approved in that way you you kind of contribute and then the other side of it is we have the Hub which is where people share their models and their data sets and more recently like their demos and I think here um I expect in the in the coming year there to be like quite a large explosion of um like models and data sets around um the building blocks of chat GPT and we're already starting to see the start of that with um this llama model I don't know if you if you've heard of this but uh yeah Facebook released a model called llama and um uh shortly after that some researchers at Stanford uh generated a data set called alpaca which is essentially a synthetic um instructions and demonstrations from cha GPT or one of the one of the sort of instruct models from open AI and then people have now gone berserk training lots of llama models on these types of data sets because the quality of the instructions or the generations of these models is very good so when you chat with one of these like alpaca models it's almost like chatting to chat GPT I mean it's inferior in many ways but it gives you that kind of magic feeling of like oh this thing kind of understands a little bit what I'm talking about so um one thing I could imagine people contributing to would be creating kind of like these types of models potentially in non-english languages so um you know can you create like a kind of like an alpaca type model in German Spanish Italian I think that's where a lot of value is because you know many of these like uh sort of non-english languages I mean they still need to do NLP but most of the resources focus on on English and then the other side of that um is around like community so hiking face typically has several events um throughout the year ranging from you know training on tpus with diffusion models to um you know conventional Like You Know audio type things and taking part in those is often one way to contribute and learn usually quite a new set of skills um if you want to for example train on tpus you get to learn how that works I don't know if it answers exactly what you had in mind um was it roughly what you were thinking of were you thinking more broadly like what does the future look like when we've got you know chat GPT writing our code for us yeah it was definitely part of the question I was pretty like the framework part it was pretty informative for me to also not to focus on one and this is a great piece of advice and what about other like programming languages is python enough or should we try to learn Cuda or you know tree ton or C plus plus or or maybe JavaScript right to to be better at writing like editing spaces you know maybe a little bit on that yeah so so I think my advice is going to be biased like a little advice um so I only know like maybe two and a half languages so python C plus plus and a bit of java so I I have no idea about front end and I've always stayed away from it um but um from what I can tell um maybe the the most relevant language uh for machine learning outside of python seems to be rust at the moment um so like I don't know if you're familiar with rust but roughly speaking it's it's like a very uh performant language which avoids some of the you know gotchas in C plus with memory um and for example large parts of the Huggy piece ecosystem are written in Rust for example the tokenizers library its back end is rust and um some of the tooling that we're building um in the next very few months is going to involve some elements of rust as well if you want to scale this like RL um stuff to like really large models you start having to worry about like asynchronous um kind of um messaging and yeah but that rust is the thing to do so that would be maybe the other thing to think about but my personal experience has been learning a language for the sake of it for me I just don't remember a stuff like it just goes in and then one week later I've forgotten it so ideally you would like to find a project or a need where you absolutely need to do that and then that really you know is the forcing function to um to master it um but yeah I I feel python is going to stay for some time and I don't think you should worry maybe too much about the the type of language itself unless you really have a need for high performance okay thank you sorry yeah so so uh thank you very much for your detailed answers um I was wondering just so piggyback on on this question um is Tony ml this idea of sort of trying to embed machine learning in systems where you don't have access to lots of gpus or you want to to be running these models on a consumer Hardware is it going to be a focus for for hugging face because I've seen that recently you mentioned alpaca I've seen the low rank adaptation method where you can sort of train these models on a single GPU there's also a c plusification of animals like llama for example whisper.cpd I think that people are really interested in these things because they want to run their models on their own like crappy Hardware I mean my Hardware is very crappy and we have a single Market I don't have any gpus or anything and so I was wondering if it would become a focus for hugging Pace as well or yeah yeah I I think um maybe not a big Focus but definitely something that um some of our team members are looking at so um one of my colleagues he he created or he piggybacked on this llama CPP and he created like a bloom CPP um Bloom is like one of these language models that we have I I kind of hope that um all these like you know like llama.cpp things become the new Doom like can you run Doom you know there's this joke right can you run doom on like you know your your phone can you run it on your you know pregnancy test type stuff and um maybe maybe you know the next Frontier is like can you run some language model on these devices um Apple watch or something like this exactly exactly and another area where there's definitely been a lot of work done um at hugging face has been around uh diffusion models so um again you can always you know do inference on some big fancy GPU for like stable diffusion but people typically want these on device um because you know maybe you don't want to have to send your photos somewhere um and maybe you just want it to run on your laptop right like you just don't want to have to keep you know paying for these like costs um and so they've been quite some work from one of my colleagues with Pedro on you know making this work on like Apple's neural engine and stuff like that so I think there's like quite exciting elements there um I think the the other question of like um efficient fun tuning um I I also really think this is going to be a big topic this year um so just for everyone else who's like not sure what we're talking about um in in the book right we talked about fine-tuning a lot of pre-trend models and when we do the fine-tuning we essentially would take like a pre-trained model you would slice off the head of that model which was typically you know trained for say next word prediction next token prediction and you would then take your you know task specific head and then train everything end-to-end and um one of the um kind of realizations in the last uh say a year or so has been that you know when you get to very large models this gets very very expensive so if you imagine trying to fine-tune like a 20 billion parameter model or even more um doing that whole fine-tuning process is going to require many many gpus and it's going to typically require you to do some complicated things like splitting the the model and the optimizer and all the stuff across multiple devices it gets very messy and so a lot of different researchers have come up with techniques where you basically take the pre-trained model and you freeze it in some way and then what you do is you kind of insert um these things called adapter layers so these are like little neural networks like little feed forward networks that are inserted at various parts of the of the original Transformer and then you only train those so everything else is basically kind of Frozen and you just get kind of fixed embeddings from those frozen Parts but these adapter layers are the things you train and they they get that information from the Frozen backbone and this is is very efficient it means you can train maybe one million parameters or two million parameters on top of like a seven billion parameter model and you get a really really good performance um and in particular what's really exciting about this is it allows you to essentially have one model um like your backbone model that you have to store on disk and then everything else is like an adapter that you can just switch in and out and so if you're thinking about deployment and I've got you know maybe I've got like I don't know um three different tasks like you know an identity recognition um text classification and question answering so traditionally I would need three different models like you know maybe three birds but three quite different births and with these adapter approaches you can basically have one like bird base or whatever the encoder backbone is and then you have these like adapters and these adapters are very small so for deployment purposes you just have to switch the adapter and it's only switching you know a million parameters versus having to really host and store you know maybe three seven billion parameter models um and just like this week I was experimenting how far can you push this and I was able to train or fine-tune like the Llama 65 billion parameter model on on one um a100 so okay it's a big fancy GPU with 80 gigabytes of vram but still the fact that you can do a 65 billion parameter model on one GPU um you know hopefully over time the price gets cheaper and then you know this will be kind of commodity Hardware so I think that's the most exciting area for practitioners this year so what was the task you you find you did for instruction tuning so I'm trying to create uh instruction models like everyone else there okay wow okay yeah thank you in fact there's a really cool paper I'll um see if I can find it um so there's a nice survey um of all these methods let me just dig it up and it's got uh let's see it basically gives a very nice overview of the last um like kind of a couple of years of these efficient techniques and um at hunting face one of my colleagues has created a library called um pep and this Library basically integrates with Transformers and it does all of these like fancy techniques where you um basically you know only train a small number of parameters and um yeah if you if you want to try it yourself it's it's quite nice great thank you yeah so it's already possible then to do this in her episode I didn't know oh that's great that's great because I was wondering I was just thinking maybe it's going to become more and more common so maybe Hogan confetti would be would be great to have something like this on her face I didn't know it was impossible you guys are so so quick at implementing things it's crazy well got a question from the chat Sam asked I was wondering if you could comment on fine tuning llms versus using zero shot and few shot that's a great question um so I would say before chat gbt um I would have said more or less do the following um if you have access to a land like a large language model maybe as your Baseline do some like zero shot or a few shot um learning um you know if you're thinking about how do you solve a data science project and then that gives you like a rough idea of performance and then if you collect some data and label it then do language like the fine tuning um and the reason I would have said it that way is because like uh the the fine tuning with you know a few thousand label examples typically gives you much better performance than on few shop and one of the problems with like fuse shot has traditionally been that you have to frame The Prompt in a certain way so um I don't know if anyone has done this themselves but last year I worked in a project with this and it was a nightmare like you know if you change one word in the prompt uh you've got different results and you know all of these like metrics that measure performance and models they usually have very large uncertainty because of that and so this whole like prompt engineering Stuff felt very unsatisfying and so fine-tuning is like typically you know the thing you want to do it requires the least amount of like you know screwing around with that however chat TPT and I think gpt4 um changes a little bit the psychology of like what's possible with zero shot um primarily because when you talk to say chat gbt a lot of the time the first thing it gives you is is quite good I mean if you say how do I do X how do I do this you know it does make mistakes but it is quite good and so I feel like the the era of like um let's say fine-tuning language models versus um just using like an off-the-shelf API um is going to largely depend on like you know does your company let you send your data out elsewhere um and you know if not then fine-tuning is probably the thing to do um however the the open source Community I think has kind of traditionally lagged a bit behind you know open Ai and other companies in producing like very high quality language models and llama I think is already showing us that you know it is possible to get very high quality um sort of decent sized language models and so this this thing about having to use the private API may also not become so important in the next year or so so to sum up uh I would still use like zero shot few shot as a baseline um but I think for most Enterprise applications fine-tuning um is probably the way to go um if you want to solve a very domain specific task thank you so much for the answer really good answer um you touched a little bit on private data I was wondering if you have any thoughts on the less spoken issues like privacy ethics copyright issues um is parking phase working on those issues yeah so we're working on it on a of a team more like a policy side of things um so we don't yet have any um like specific tools to help around you know privacy and and you know other questions that are important around you know uh sort of like the the biases of some of these language models um so on the policy side um we have um people like Meg Mitchell I don't know if some people have heard of her she she's a fairly prominent ethics researcher who used to be at Google and is now at hugging face and um she she leads a lot of the efforts around how do we at hugging face set up um roughly good practices um as a kind of indicator for the community to sort of adopt and in concrete terms um that the way that we think about this is that um a lot of the discourse around let's say uh the release of language models um is is often trained as like well our language models are too dangerous so open air for example says their language their technology is so powerful that if we made it totally accessible then you know um you know who knows what would happen and so the way we're thinking about that at homeface is to think about how do you still retain like open source models but allow some sort of gating on them so that the the people who release the models have some control over its distribution um and this was you know partly from from Mega is one of her ideas and um around the like say copyright and stuff that I think is a still an unsolved question in the sense that like at hugging face um the best we we can do is kind of guide um other organizations on you know for example stability worked with us around like the release of stable diffusion and there there's a lot of issues around like first of all you know what does it generate and what was it trained on and so we we usually guide organizations on like thinking about you know what type of data you may not want to include in the training of your models um and I would say on the Privacy side um yet we're not I would say doing anything concrete around like I don't know if you're thinking about things like you know polymorphic like uh you know encryption and stuff a lot of models but it's more more like you know in a general sense like we're we're sort of just acting as an advisory as advisory role um but yeah I'm not an expert in that stuff so as you can probably tell my answer it wasn't uh the most uh uh in-depth one well thank you so much for your answer I think uh it's a big field people should uh consider I think that the most important lesson I've learned from um sort of interacting with the ethics team at hiking piece is that a lot of the decisions that you make as an engineer um are often focused exclusively on the performance of the model so um you say to yourself okay what could I do to make the model better okay I could get this type of data and just throw it in and then you know see if it works and um one of the kind of lessons from from working with the ethics team has been to have a more considered approach which is that if you define upfront what are your kind of guiding principles of how you will collect data it already frames your mind a little bit differently to say okay maybe I won't scrape this whole bunch of you know sort of dubious content just to get better performance I should maybe find alternative Solutions um and that framing I think has been quite helpful um internally in time in terms of trying to think about you know not just doing what everyone else does and doing something a bit different that makes sense thanks anyone else have questions okay I'll keep asking then things are happening so fast how do I keep up just like so much things are happening every week it's crazy tell me about it um so so yeah I don't have any I think Super actionable advice here um so like everyone I think we we suffer from this now especially because um the rate of progress on on many domains is happening incredibly fast so I think the release of like stable diffusion kind of kick-started this accelerated Pace where you know every day if you're on social media you would see like you know the next model being released with you know even more images of astronauts riding horses and stuff like this and I think there's a danger of getting first of all like a very bad dose of promo and feeling that like you know there's this amazing world of like rapid progress happening and you're you know stuck doing I don't know Transformers 1.0 and you're like oh the world is moving um and the other side of it is that um uh like it's hard to predict where like you know the next thing will happen and so if you sort of follow the hype you you chase the you end up with a problem that you maybe get a very shallow understanding of what's really happening so my own personal approach has just to ignore it basically um as in I I picked a domain that I thought I could contribute to which was this blend of NLP and now reinforcement learning and that's been my personal focus and part of that has now involved basically yeah not looking at Twitter very much and essentially just trying to focus on on making my own Project work um however you can't fully ignore stuff because it impacts you know your your the world and your work and so um if you want to keep up uh I guess there's a few ways you can do it one is um join a community that is active and you know friendly and all those things and you know there are many of them for example of course hugging Pace has its own Discord server there's lots of people sharing ideas and news um but there's also other ones like Fast AI has a very healthy community and there you can learn a lot of like deep learning techniques um and the other thing I want to mention is like when I was working as a data scientist um there's like like there's like hype stuff you see on Twitter and then there's reality and reality is like um much more like pragmatic about like for example if you're trying to train like I don't know a classifier um for you know classifying documents like inbound requests from customers or something um you may just end up having a much better job by just having like a baseline with like I don't know bag of words and IBS and this will be far more easy to deploy and debug in production than like you know some you know 175 billion parameter language model and so I think there's a tendency of people to say oh I have to get the next shiny thing um but being a very good data scientist in my kind of experience is being as simple as possible um and so you know if that's what your profession is then I think focusing on those kind of fundamentals is where you have the biggest impact um and so yeah balancing fomo versus impact is is the hard part but um yeah everyone has to own me exactly the same comments I mean even though that there's a lot of shiny things out there or problems are kind of Odin we can apply like the current techniques or old techniques and and so of then like so yeah exactly yeah I think one of the most impressive data scientists I ever had the pleasure of working with he could solve almost all of our problems with SQL like he was an SQL Wizard and it was incredible like you know I was trying to do XG boost and stuff and he was like ah nah let's just do SQL and you know I know it sounds like a running joke in data science teams but it really makes your life less painful when you're not trying to debug like a massive neural network in production that is so true reality is so different from what's in the Twitter feed and in the news exactly well do you think what would happen the next five years or even one year maybe next month oh yeah that's a very tough question so yes yeah yeah like I think what's really hard here right is if you had asked me so so for example right when we finished the book um we kind of ended with a glimpse of like audio and vision Transformers and and you know if you had asked me oh there's going to be this enormous explosion of like diffusion models I would not have predicted that like it's like a thing where unless you're you know one of maybe 10 100 people in the world who was following that like one research group in Germany it would have been very hard to predict that um and I think the same is also kind of true for things like um chat GPT in a way I feel like the technology or the underlying fundamentals of what goes into that open AI has been doing for quite some time but the interface and the kind of impressive engineering that goes into like serving this to like hundreds of millions of people um is is the transformative effect and again you couldn't have predicted it right like I think up until chapter everyone was like oh yeah you know open AI they have these apis but you know like is it so cool and now you've got this kind of thing that you know when I go in the train people are speaking about it in German and like it's like just crossing languages right so I don't want to make any concrete predictions because they'll be wrong um but the kind of let's say obvious ones I think is this thing I mentioned at the start of progressing towards more modalities um so one of like the very cool things um that has you know been emerging especially with diffusion models is that if you combine for example text and images you can get you know far more kind of creative expressivity um around what you're trying to do and so I could imagine that um you know gbt4 is already kind of multimodal and we'll start to see I think from the open source Community um a strong push to you know not just train language models but to train multimodal models um so that's something I think is going to definitely happen and the other area where I think we might see some interesting progress is is around um audio and and you know music synthesis um and again diffusion models I think really showed um at least to me what's possible um with like creating synthetic music um you know just from either text prompts or from other things you can you know almost craft now songs just by saying give me a song a rock song in the style of blah and you know about a beach and then it can generate that which is quite cool um so those are like the sort of obvious ones I think the the less obvious ones um might be around like what's the ceiling of like uh capability um from scaling up or you know adding these extra modalities so um at the moment you can sort of see that these language models have some rudimentary form of like reasoning and I mean okay you can debate the word reasoning but they have a rudimentary form of it and they're also I think quite um kind of exhibiting rudimentary forms of like theory of mind and stuff like this and the question is like will we see in the next five years something that um like a system that is capable of going Beyond this kind of like um I don't know uh I give you a question it gives you an answer but something that is potentially more like self I know that that's a word like can generate something maybe hypotheses and then test them um and that that seems to be so far the thing that's kind of missing of like you know when we wake up in the day usually you say oh what I'm going to do today and you have this whole kind of like you know imagination of things you can imagine you will do and then you sort of reason about them um I don't think our language models can do that yet and that would be I think a big uh change I mean if we got to the point where you could do maybe like fundamental science with like language models um that would be I think a very big change and you know the rate of progress is so fast who knows it could happen in the next five years um uh but uh anything else I say would just be wrong so it's probably better not to make predictions on camera I love it I love it love it so much um we're out on time anyone has burning questions you want to ask yeah maybe about the reasoning thing uh because I'm really interested in this uh I'm sort of transitioning towards uh sort of Transformer based program synthesis these days uh which is almost the same two sides of the same coin for me already like automated tier improving and if you can generate programs you can also generate theorems because really theorems and programs are the same things uh there's this thing called this Curry Howard correspondence in in type Theory and uh and I'm just I'm just completely like I'm a bit drowned in this so many like so many attempts everywhere I'm not exactly sure what to where to look at and I'm not exactly sure what's the actual progress has been made because some people are saying oh to actually reach reasoning I mean I mean have good capacity of reasoning for these models uh you need to go through um the plug-in approach we've seen it with the charging plugins where I can sort of call I don't know like a few improvers or stats covers and yeah and some other people are saying no no you should actually to modify the architecture and sort of there should be a way to modify the architecture and force the network as sort of a neurosymbolic approach where I say your network should be able to do simple equation as well and I don't know I don't know uh yeah what what are your thoughts on this uh have you have thought about that is a bit about this um I wouldn't say in any sort of deep sense um I I think probably the most let's say promising approaches today so one of the big limitations right of of language models um tends to be right now the the fact that they hallucinate um facts and stuff right like you wouldn't want to rely on Chachi PT for like life and death you know facts like how do I handle a snake bite or something you know maybe like suck the blood out or something um and the kind of conventional way of thinking about this is like okay maybe then you should couple it to like a database or a search engine or something like this and that's going in the direction of tools um and these plugins um now whether that is like so that definitely assists with um with for example fact-based reasoning um in the sense that if you ask someone you know who is the president of the US that they will do some kind of like reasoning lookup you know it used to be Donald Trump now as Joe Biden and these models seem to do something analogous to that um but uh the the sort of deeper question of like um do these models like reason about say mathematics um in in similar ways to us um I think no one really knows like if that's the case and um there's like a lot of interesting work I guess you may have heard of Tim gowers he's a quite famous mathematician who's working on my computer improving and um there he sort of talks about the fact that like even if you have language models that can generate proofs um if those proofs can't be kind of represented in a human intelligible form they're kind of practically useless um for people and so probably the the main challenge here will be figuring out how do you get language models to kind of explain their reasoning um in ways that we understand um and can make use of because and I I don't think that's like a really a solid problem um but there is one paper I'll share which I found quite fascinating um oh thank you there's a paper about evaluating um GPT core or Microsoft and um in this paper they um they they looked at like different elements of like let's say reasoning and intelligence of like how you might evaluate things and and one of them that they have is this theory of Mind evaluation and also mathematical reasoning and those those you know might be like precursors to thinking about how do we you know evaluate these models Beyond you know the conventional benchmarks and stuff yeah the thing that frustrates me about gpt4 is that they didn't see anything about the architecture so for me we're not doing science anymore it's just a product for example is a product right yeah exactly they're frustrated I see some benchmarks but I don't know what's going on and it's an even worse right like um one of the big challenges with all of these like pre-trained models is they often involve scraping the whole internet really in its entirety and uh people showed shortly after gpd4 that like um on one of the evaluations which was like I think a code like code type thing um essentially it got like great scores until 2021 and then 2022 completely failed um and so that shows that the models are being trained on the same data that they're evaluated on and open AI has done usually quite good job of saying trying to estimate the contamination between the train and evaluation sets but even then at some point you're not you know fully fully capturing that and so I guess yeah these benchmarks get a bit harder and harder to sustain as you know they keep getting scraped yes that's the thing that's why some some of these results I suspect are red herrings really because like especially the moth section so I I assuming the people you I really scouted the paper I was with I geeked out on this for the past few uh you know days or so and uh and the exercise the math exercise they gave is a classic sort of exactly the grad exercise and I'm sure there's a solution on the internet to be success I'm absolutely positive because I could even remember myself the solution of the exercise so I would suspect this data contamination when it comes to mouth abilities like uh for these things um like Infinity of primes as a poem is quite nice I mean this is really nice but but as the poem is is the impressive thing the Infinity of probe is obviously yeah obviously there's thousands of proofs on the internet that's right yeah then I agree that's that's very uh yeah but that data contamination needles because I I tried on my own I thought okay uh I tried in an area that's not very well spread on the Internet it's called category Theory I thought okay uh so I I checked that GPT dpt4 knew the definitions I was talking about and he knew it perfectly but in your legit with a Wikipedia definition no problem whatsoever and then I tried to make half GPT uh prove a very simple fact about category Theory based on these definitions but I'm sure I I don't think these factors are not true of course but I don't think it's on the internet because for experts this box is called folklore so it's basically mathematicians don't write these facts on the internet because when you get to the point where you understand the definition the fact is obvious so nobody would write it because you look like you look like a beginner if you write this on the internet really that's it so I thought okay I have to ask control questions because then it's super simple to prove but it's not it's probably not going to be on the data sets and it completely was completely wrong it was completely wrong like very very basic reasoning was absolutely wrong and I thought okay okay so maybe maybe actually there's a lot of contamination about the you know the traditional sort of Benchmark we see on on Master exams for these things exactly yeah I totally agree and I think like a future where that is improved significantly um as a former scientist that would be very cool like imagine having like you know a lot of the time in research you you talk to people and you have a Blackboard and you're writing an equation then you're discussing things and stuff like that and I think having that kind of pair programming but now it's like a pair researcher yeah yeah I think that would be very very powerful um but we're amazing we're far away I agree even even having heuristics like I'm not even asking for rigorous proofs very good because I think it's a picture of mathematicians right but even heuristics having some intuitions sometimes the problem because map is as much an Arts as it is a science like it's it's really about creativity and sometimes you just can't think about a creative way of getting out of your of uh you know whatever whatever situation you're in and uh and I think the platform which model should be able to help with this but again like you can't you can't be too creative you have to be creative within the bounds of logic and I think they're really bad at respecting this so so far awesome wow I love your guys conversation uh fascinating I'm wondering uh Louis do you have a few more minutes to answer the final last question from the chat so smaller companies now prefer API plugins than spending time doing all the other work I guess so as an NLP engineer what areas should we focus on yeah that's a really good question um so I think the general lessons um of like say you know the previous approach versus like say potentially this new API based approach um I I don't know I I sort of feel that as long as your business is in a domain that is like sufficiently specialized um the the utility of these general purpose apis is going to perhaps have some some limited applicability um and in any case um I think focusing on the quality of the data that is specific to your domain um will always be valuable because even if you have some super duper API um generally speaking fine-tuning those models on your own data will give you even better performance um so I feel like focusing on those fundamentals around data quality and um you know how do you measure the performance of the thing you're trying to solve um that's still going to be valuable I I I would be surprised if that goes away um entirely um and probably as an NLP practitioner the field is kind of transitioning um to an Era where um Hardware is getting commoditized um you know a bit faster than than normal so um today right it's kind of well let me rephrase it a few years ago training Bert was like oh my God like impossible only Google can do that um blah blah blah and you can now train Bert roughly to the same performance as Google I think for like 30 bucks um you know if you're if you're if you're very clever with like you know um sort of efficient uh Computing and so I could imagine that a future where you don't have like a super cluster of gpus but you might have like four gpus but you have a very efficient way of training on those gpus would be still valuable so um my sort of General thing is like don't don't Maybe you know go all in on apis yet it's probably a bit early for that but really understanding how to do like efficient training and I think pre-training is probably going to become more commonplace um I have I might be wrong but I have a hunch that like you know learning how to do this on a decent budget it will become more common for for companies because for example like in Switzerland um we have like languages that probably don't even exist very well in like you know some of these apis like Swiss German and Vermont and stuff and again you could try to hack around it but eventually you might want to train your own Swiss model if it becomes sufficiently cost effective and you know you've got the data for it a very very generic advice but yeah just focus on it focus on I think engineering um I think that will that will generally serve you well I believe even the basic knowledge on NLP can help because usually you're a stakeholders they have no idea how to solve that problem so if you know the basics then you can guide and explain it so yeah that's one of the things I I really found very good about the book because he gave me so much Clarity he gave me so much tools to to know address these problems that should be really simple but yeah they were not for me because I didn't know how to deal with them so yeah I think that's also available to know awesome we're at time thank you so much Lewis I feel like I've learned so much thanks for having me it's been a really fun discussion and thanks for all the questions it's uh it's a real pleasure when people are super engaged and asking hard stuff so yeah we love your book I feel like I need to read it I'm a third time to fully understand everything but thank you so much I should probably read it one day because you write a book you don't want to look at it for a while okay um yeah I will pause the video online if anyone feels uncomfortable about showing your face or your voice please let me know and the ways I will send you the link when the video is up great thank you thank you very much thank you very much thank you thank you