Graph Neural Network Project Update! (I'm coding GAT from scratch)

Key Takeaways

what's up folks so this video is going to be an update and not a classical video on my channel where i'm just covering some research paper so basically um what i've been doing is over the last seven days i think i've been uh coding this graph attention network or get for short from scratch and i actually started somewhere around new year's time and i just continued now i'll be just skipping the the classical research paper overview because for those of you who know me you know that i do i mix a lot of theoretical work uh but i also do a lot of hands-on like deploring projects and stuff so now the the the the time has come pretty much in my in my in my like uh workflow to to just kind of pause the the theory a little bit and focus on encoding as you probably know uh like my github profile has a lot of projects those are some of the the projects i've been developing throughout the last year make sure to check them out i've got a neural style transfer on my github you can play with that i've implemented both cadence's approach so that's the original optimization based approach i've also got johnson's approach which is the fast forward like just using cnn to stylize your images i've got us even some really cool uh projects where you can just basically segment yourself and stylize everything around or style as yourself whatever so that's cool check it out as well then i've got deep dream which was really awesome artistic project and you can learn a lot about cnns and deep learning in general by doing this project so make sure to check that one as well then i've been developing generative reseller networks as well so i did like develop the original one i did develop the conditional again and finally the dc game which kind of started the cambrian explosion uh in the world of generative research networks so that one is really cool and then continuing i was developing uh the transform i was working on transformers and i actually open source the original was funny transformer so i really like to think that that project is a super super good place to uh to start supplementing the paper and understanding how it actually works so every single detail so we got it's got a lot of comments so yeah uh hopefully you'll find that one useful as well so now i'm working on on get graphic attention network and as any as pretty much any deep learning project you can conceptually i kind of split it into a couple parts like the first one is data then we have the the model part and finally we have the training loop and some additional visualizations uh the stuff i usually put in playground a cool playground whatever and basically uh when it comes to data because these are graphs um i've been doing some stuff that i usually don't do when i'm exploring data so i'll be doing network analysis like uh just plotting the degree distribution of my network of quaran network and some other statistics uh i did some i i found this really good tool igraph uh and i actually plotted how coral looks like and that kind of just gives me a gut feeling for for how the graph looks like and that's that's kind of useful uh you feel better about yourself nothing else you probably don't need to know all of those details to make uh to make a this project successful but it's it's really cool and you learn a lot in the process to learn different tools you learn yeah uh basically that and um finally the the model part is um so i have like four i think i have four different implementations so the first one is similar to python geometric if you're not familiar with that project it's a really super awesome uh project that was implementing most of the uh main graph uh networks so gnns so do check it out uh but the thing is my implementation is actually like almost twice as fast because i was using some specialized pi torque functions like instead of using basic numpy indexing i was using index select i was using instead of using like some suffix function i was using torsum i also don't have the overhead of having to fit my my get project inside of this message passing framework as they do so it's it's it's cleaner and it's faster so but it's only for a graph attention network so hopefully you'll find it useful i'll open source it i think next weekend so stay tuned for that one and uh the other implementations one is using a torch sparse api which is currently in beta so that's going to be fun because i'll have to develop the backprop function explicitly so it doesn't handle so those sparse functions don't handle the back prop automatically the third one is the conceptually the easiest to understand but it's computationally most inefficient so yeah so that's the gradation the training loop part is really easy because core is just a simple classification problem so nothing new there just cross entropy like some masking because we have because it's a transactive training setting and that means you you have to mask out the training notes uh but you can see the uh test notes as well as the validation notes during the the procedure so uh yeah that's nothing interesting there uh then i've got the playground where i've been uh profiling different sparse uh formats for different matrices so like uh you probably heard of some of them like cool the coordinate form format then we have dok then we have lil we have csr css cs no csc sorry uh then we've got no not css that's that's that's not the best format and then um we've got um daya and bsr so about about seven formats which sci-fi supports and what i did is i just analyzed how how fast they are and actually figure out that some of the implementations out there like gcn by thomas coupe and even get the official get by petra velitcoach used they've used a little format for some arithmetic operations like summations and and stuff like that and that's actually sub-optimal because you want to use that one as you're building up your graph as you're just modifying the sparsity structure but you don't want to use it for arithmetic operations so yeah that's a that's a fun that was a fun experimentation and also of course i love visualizing stuff i played a little bit with t-sne where i visualized the embeddings that come out from the train gap model so yeah i also play with umap that's uh that's a relatively new algorithm uh most of the people are just using t-sne as the default method but umap is also cool but in this particular case i actually found that t-sne is as good as umap so i just kind of dumped it all together and finally i want to make a shout out to my patrons to petra velichkovic from deepmind who is the aio overlord of this channel and to jimx who is the ai disciple of this channel nothing less important just want to make a shout out and thank you for for supporting my channel nothing less important the whole community i love all of your comments keep them going i just want to read a couple of them here so the first one uh great one question have you noticed that arbitrary style transfer paper has more citations than your universal style transfer one so i love those geeky ones as well as these more simple ones like these man do you ever take a holiday uh lull i appreciate all your videos so just keep them coming i love your comments and i read all of your comments and uh i'm gonna try and answer all of them hi alexa can you make a video about your background and your journey into ai ml uh yeah basically if i see more of these uh comments i'll probably make some time and create a video where i'll walk you through my journey into deep learning so i also want to make a shout out to to phil who is also supporting my channel and thank you your channel is also amazing so love to collaborate one day with you uh yeah that was pretty much it you know the drill uh subscribe hit the like button and share this video that's how you can help me uh build up this channel build up the community i want to include you as more as i can in in my in in all of this so if you have any suggestions how i can do that please write them down in the comment section and until next time keep learning deep bye