Booking.com's Vector Search Benchmark

Key Takeaways

So we created this evaluation setup with public data set. You could use anything with 100 million embeddings and we created a setup where we would send queries approximate KNN queries with and without filters because we also have a lot of use cases using filters. It can add complexity to the query. So it should also still perform well. We increased the concurrency. So we basically run bunch of experiments comparing all these different queries increasing the concurrency and we basically record the performance in terms of latency but also throughput and eventually had a nice overview of different vendors. Also our baseline is open search. So this overview gave us to make a decision which vendor stays more stable under the under very high concurrency.