Creating & Ingesting Your Own Embeddings in Weaviate | Vector Databases for Beginners | Part 7

Key Takeaways

I wanted to go over how you can create embeddings two different ways with Weaviate. So, I'm going to go over to a notebook over here in Colab. Um the first one we're going to use is bring your own vectors in Weaviate. So, with this notebook, we're going to generate embeddings with modern BERT on Hugging Face here. Um we use this model to generate embeddings and then insert them into Weaviate. So, I've already done some of the stuff up here. I've installed the Transformers library, data sets, and Weaviate. We're going to be using a data set from my colleague Shorten or from my colleague Connor um on archive paper machine learning papers. Um so, I just loaded all that cuz otherwise it would take too long during this. We're going to keep just the title and the abstract columns from this data set. And then we're going to take So, the original data set contained over 100,000 titles and abstracts. But for this demo, we're just going to take a random sample of 100 different papers. So, that's this part just taking a random sample. Um and basically at the end of that, we're going to get a data set of 100 examples. And now we need to create embeddings. And I also did this before. I was hoping to do it during this, but it took too long. So, I'm going to import the sentence Transformers library here. Set my model as the modern BERT base model. And for every example, I'm going to generate a embedding and store it under the im- beddings sort of property dictionary key um for each example. And then I'm just going to map these embeddings into our existing data set. Okay. So, when we do all this, we're going to get a data set changing it to pandas. Really hoping this works. It's always a rough time. Oh, no. I have to run all this again, I think. Oh boy. Okay. Well, we're just going to do all this quickly. >> [laughter] >> And hopefully it won't take too long. Again, we installed the Transformers library. We're going to generate embeddings with the modern BERT model from Hugging Face. We've kept 100 rows of this archive data set. And if we look at the little data set itself, you'll see [snorts] we have the title, we have the abstract, and we have the text, which is a combination of both of those. And then we have our embeddings. Um Right. So, next we're going to actually put this in Weaviate. And to do that, I'm going to use a sandbox from Weaviate Cloud. It's free, which is always good. So, I'm going to create a cluster Oh, so I'm going to go to console. that.weaviate.cloud. I have an account. I'm logged in, hopefully. Yes, I'm logged in. Um if you don't have an account, you can sign up for free. And then you click up you can go to clusters, click on this button right here, create cluster. Going to create a sandbox, and I'm going to name it embedding demo. I'm going to select Europe because I live in Europe, and then I'm going to create. Awesome. So, cluster's being prepared. And what we can do now is we can click on this connect button and we can copy all of our info. So, I'm going to copy the Weaviate URL here. I'm going to bring it back over here. And if you're doing this for real, please store these as environment variables. But for right now, I'll just paste them in there cuz it's fine if you access these. Um okay. We're also going to copy the Weaviate API key here. Bring it back over here as well. Awesome. Um and then we need to connect from here to our cloud. So, I'm going to go back over here. And what's nice about this little button is it already gives you the code. So, I'm going to uh copy this part. Go back over here. I'm going to paste this part. And hopefully when I run this, it'll work. Oh, it might not work because I don't know if it's ready yet. Okay, still being prepared. Give it a second, then. Do do do do. Put some elevator music. >> [laughter] >> Okay, while it's doing that, I'll go a little bit over the next part. So, when that works, we'll have a client is ready. And then we need to actually create our collection within Weaviate to store our vectors that we just made in. So, I imported all the Weaviate stuff up here, and I'm going to define my collection name as bring your own vectors collection, BYOV collection. Um and then this is just if I already have a collection, delete it. So, then I'm going to initialize a collection by client.collections.create client.collections.create. My collection name will be the collection name. And then I have to set a vectorizer config. And we have our own vectors for this one, so we actually want to set the config to none. So, then I'm going to do wvc. config. dot configure. configure.vectorizer. dot none. And this is because we already have our vectors. So, we don't want to actually vectorize anything within here. Is my cluster ready yet? Yeah. Okay. Cluster's ready. Hopefully this should work now. Yeah. So, client is ready. True. And then we can initialize our collection. Nice. Run. Um next thing we're going to do is we're going to insert all our data into this collection. So, I've defined properties here. We have a text property and a title property. Um and then I can insert text, title, and also our embeddings into the collection. So, by using this vector property. And I'm using insert many. And if I run this, [clears throat] it will put all of our data into the collection. If we go back into the cloud, reload this. Hm. Reload this again. Um go to the explorer. And our embedding demo and our bring your own vector collection, we can actually see all of our stuff that's in Weaviate already. So, we have our vectors here. Nice. We have our text, we have our title. Great. So, everything looks good in here.