In this video, we’ll take a look at vector embeddings based semantic search, a powerful technique for finding relevant information in large text datasets. We’ll explore the fundamentals of natural language processing, explain how vector embeddings represent words and phrases in high-dimensional space, and demonstrate how semantic search can effectively identify related content by measuring the similarity between these embeddings.
Links:
Blog – https://partee.io/2022/08/11/vector-embeddings/
OpenAI API – https://platform.openai.com/docs/guides/embeddings/limitations-risks
0:00 – Introduction
1:50 – OpenAI Documentation & common use cases
4:48 – VSCode, loading libraries, utils, basics of vectors
8:03 – Generate embeddings for list of words
9:56 – Cosine similarity function
11:49 – Vectors in 3D space explanation
14:07 – Cosine similarity applied to dataframe
14:35 – Working with longer, realistic documents/text
16:38 – OpenAI synthetic document/knowledge-base generation
17:15 – Semantic search on knowledge-base
Socials:
https://www.linkedin.com/in/pratheekdevaraj/
https://instagram.com/patdevaraj?igshid=NTc4MTIwNjQ2YQ==
