We build a multi-modal hybrid search engine for ecommerce using OpenAI’s CLIP, BM25, Pinecone vector database, and Python. The search engine processes text and image-based queries and can produce better results than traditional methods.
The search engine allows users to search and retrieve data using both text and visual queries, which is especially useful in e-commerce domains where users have a range of search queries, from specific product searches to image-based searches for related items.
By using CLIP and BM25, the search engine can process both text and image-based queries, providing users with a comprehensive search experience. Additionally, Pinecone vector database and Python allow for easy indexing, storage, and retrieval of data, making it possible to handle large volumes of data in real time.
Example notebook:
https://github.com/pinecone-io/examples/blob/master/learn/search/hybrid-search/ecommerce-search/ecommerce-search.ipynb
AI Dev Studio:
https://aurelio.ai/
Discord:
https://discord.gg/c5QtDB9RAP
70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5
Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership
00:00 Multi-modal hybrid search
01:05 Multi-modal hybrid search in e-commerce
05:14 How do we construct multi-modal embeddings
07:05 Difference between sparse and dense vectors
09:43 E-commerce search in Python
11:11 Connect to Pinecone vector db
12:04 Creating a Pinecone index
13:45 Data preparation
16:32 Creating BM25 sparse vectors
19:33 Creating dense vectors with sentence transformers
20:26 Indexing everything in Pinecone
24:41 Making hybrid queries
26:01 Mixing dense vs sparse with alpha
32:11 Adding product metadata filtering
34:13 Final thoughts on search
