Semantic Search¶
Let's work through the semantic search implementation for the Advanced Assistant Copilot, focusing on the cosine similarity search algorithm and setting up the necessary AL code and data structures in Dynamics 365 Business Central.
Semantic vs Keyword Search¶
Semantic search and keyword search are two fundamentally different approaches to retrieving information from a database or search engine. Let's explore how they differ, using examples to illustrate the distinctions.
Keyword Search¶
Definition: Keyword search is the traditional form of search that retrieves information based on the exact match of keywords within the text data. It does not account for the context or meaning of the words.
How it Works: The search engine looks for precise matches of the user's input keywords in the text. If the keywords appear in the document or database entry, that entry is considered a match.
Example:
- Search Query: "apple"
- Results: Documents containing the word "apple," which could be about the fruit or the technology company, depending on other keywords in the search or the focus of the database. Without additional context, both types of articles may appear in the results.
Limitations:
- Context-Agnostic: Does not understand user intent or the context of the words. For instance, keyword searches can't distinguish between Apple the company and apple the fruit without additional clarifying keywords.
- Rigid: Misspellings or synonyms (like "smartphone" for "iPhone") might not retrieve relevant results unless precisely matched.
Semantic Search¶
Definition: Semantic search uses the underlying meaning and context of words, not just the words themselves, to produce more relevant search results.
How it Works: Semantic search technologies use natural language processing (NLP) to understand the intent and contextual meaning of words. They often utilize machine learning models to interpret queries and document contents, considering synonyms, related terms, and even the sentence structure to find the best matches.
Example:
- Search Query: "apple"
- Results: If the database or search engine has context about technology topics, the search might prioritize articles about Apple Inc., especially if recent searches or user profiles suggest a tech focus. Conversely, in a culinary context, it might return results about the fruit.
Advantages:
- Contextual Understanding: Can discern user intent and the context around search terms, providing more relevant results based on the meaning rather than just the presence of specific words.
- Flexible: Can handle variations in language such as synonyms, related concepts, and even some natural language queries. For example, searching for "tips for better smartphone photos" might return results that discuss "improving iPhone photography" because it understands the relationship between these concepts.
Graph Representation¶
If we present embeddings as a graph, we can see how semantically similar items are close to each other. For example, the following video shows how sessions about "ethics" are close to each other.
Implement Semantic Search¶
1. Create Search Results Table¶
First, we need a temporary table to store the results of our semantic search. This table will hold the session codes, their respective cosine similarity scores (described later), and the payload for reference.
Open the "src\6-AdvancedAssistantCopilot\SemanticSearch\SemanticSearchResult.Table.al" file and add the following code:
fields
{
field(1; "Session Code"; Code[20])
{
Caption = 'Session Code';
}
field(2; "Cosine Similarity"; Decimal)
{
Caption = 'Cosine Similarity';
DecimalPlaces = 0:9;
}
field(11; "Payload"; Text[2048])
{
Caption = 'Payload';
}
}
keys
{
key(PK; "Session Code")
{
Clustered = true;
}
key(SK; "Cosine Similarity")
{
Clustered = false;
}
}
- Session Code: This is a reference to the session in the main session table.
- Cosine Similarity: This decimal field stores the cosine similarity score between the user’s query vector and the session's vector.
- Payload: This text field stores a snippet or the entire content related to the session for quick reference.
2. Switch Assistant to Semantic Search¶
Now, let's implement the AL code that will handle the semantic search logic, including vector comparison using the cosine similarity formula.
Open the "src\6-AdvancedAssistantCopilot\SemanticSearch\SemanticSearchImpl.Codeunit.al" file and add the following code:
[EventSubscriber(ObjectType::Codeunit, Codeunit::"GPT GetSessionDetailsByName", OnBeforeSearchSessionUsingKeyword, '', false, false)]
local procedure SearchSessionUsingSemanticSearch(var Session: Record "GPT Session"; SearchText: Text; var Handled: Boolean)
var
SemanticSearchResult: Record "GPT Semantic Search Result";
begin
if not CheckIfEmbeddingsExist() then
exit;
SemanticSearch(SemanticSearchResult, SearchText, 0.77);
SelectOnlyRelevantSessions(SemanticSearchResult, Session);
Handled := true;
end;
For Business Central 24.0 or 24.1, you can use the following event subscriber:
[EventSubscriber(ObjectType::Codeunit, Codeunit::"GPT Event Assist. Tools Impl.", OnBeforeSearchSessionUsingKeyword, '', false, false)]
This code will trigger the semantic search process when the copilot receives a search query. Then it will deactivate the keyword search and only show the most relevant sessions based on the cosine similarity threshold.
3. Semantic Search Logic¶
- First, we need to convert the user's search query into a vector using the embeddings.
- Then, we calculate the cosine similarity between this vector and all the vectors stored in the system to find the most relevant sessions.
- Finally, we filter the results based on a minimum threshold to ensure only highly relevant sessions are displayed.
procedure SemanticSearch(var SemanticSearchResult: Record "GPT Semantic Search Result"; SearchText: Text; MinimumCosineSimilarity: Decimal)
var
SearchTextVector: List of [Decimal];
EmbeddingImpl: Codeunit "GPT Session Embedding Impl.";
begin
SearchTextVector := EmbeddingImpl.ConvertJsonArrayToListOfDecimals(EmbeddingImpl.GetAzureOpenAIEmbeddings(SearchText));
CalculateCosineSimilarityForEmbeddings(SemanticSearchResult, SearchTextVector);
SemanticSearchResult.SetCurrentKey("Cosine Similarity");
SemanticSearchResult.Ascending(false);
SemanticSearchResult.SetFilter("Cosine Similarity", '>=' + Format(MinimumCosineSimilarity));
end;
4. About Cosine Similarity¶
Cosine Similarity is a way to figure out how similar two documents are by looking at the angle between their content represented as vectors in space. Imagine each document as an arrow pointing in a certain direction; the closer these arrows point in the same direction, the more similar the documents are. Cosine similarity calculates the cosine of the angle between these two arrows. If the value is close to 1, it means the angle is small and the documents are very similar, even if they are far apart in terms of actual content length or word count. Learn More
Formula for Cosine Similarity:
where:
- \( A \) and \( B \) are vectors.
- \( \cdot \) denotes the dot product of the vectors.
- \( \|A\| \) and \( \|B\| \) are the Euclidean norms (or magnitudes) of the vectors.
This formula will calculate how similar the user's query vector is to the vectors stored in the system, thus allowing us to retrieve the most relevant session information based on semantic content rather than just keyword matching.
When using text-embedding-ada-002, the vectors are already normalized, so the cosine similarity calculation is simplified to the dot product of the two vectors.
5. Find Relevant Records¶
This procedure calculates the cosine similarity for each session embedding and the user's query vector, saving the results in the temporary table for further processing.
procedure CalculateCosineSimilarityForEmbeddings(var SemanticSearchResult: Record "GPT Semantic Search Result"; SearchTextVector: List of [Decimal])
var
Embedding: Record "GPT Session Embedding";
CosineSimilarity: Decimal;
begin
SemanticSearchResult.Reset();
SemanticSearchResult.DeleteAll();
if Embedding.FindSet() then
repeat
CosineSimilarity := CalculateCosineSimilarity(Embedding, SearchTextVector);
SaveSearchResult(SemanticSearchResult, Embedding, CosineSimilarity);
until Embedding.Next = 0;
end;
6. Cosine Similarity Calculation¶
Now, it's where the magic happens! This procedure calculates the cosine similarity between the session embeddings and the user's query vector.
procedure CalculateCosineSimilarity(var Embedding: Record "GPT Session Embedding"; SearchTextVector: List of [Decimal]) CosineSimilarity: Decimal
var
Vector: Record "GPT Session Embedding Vector";
SearchTextVectorValue: Decimal;
begin
Vector.SetRange("Session Code", Embedding."Session Code");
if Vector.FindSet() then
repeat
SearchTextVector.Get(Vector.VectorId, SearchTextVectorValue);
CosineSimilarity += Vector.VectorValue * SearchTextVectorValue;
until Vector.Next = 0;
end;
Performance Considerations
Calculating cosine similarity can be computationally intensive, especially with large datasets. Considering you have 100 sessions, each user query will result in 153,600 calculations (1536 dimensions x 100 sessions). For larger datasets, you might need to optimize this process or consider using specialized vector databases like Azure Vector Search or Qdrant to handle the load efficiently.
7. Save Search Results¶
Now, we save the search results in the temporary table for further processing and filtering.
local procedure SaveSearchResult(var SemanticSearchResult: Record "GPT Semantic Search Result"; var Embedding: Record "GPT Session Embedding"; CosineSimilarity: Decimal)
begin
SemanticSearchResult.Init();
SemanticSearchResult.TransferFields(Embedding);
SemanticSearchResult."Cosine Similarity" := CosineSimilarity;
SemanticSearchResult.Insert();
end;
8. Select Only Relevant¶
Finally, we filter the search results based on the cosine similarity threshold to display only the most relevant sessions to the user.
procedure SelectOnlyRelevantSessions(var SemanticSearchResult: Record "GPT Semantic Search Result"; var Session: Record "GPT Session")
begin
if SemanticSearchResult.FindSet() then
repeat
Session.Get(SemanticSearchResult."Session Code");
Session.Mark(true);
until SemanticSearchResult.Next = 0;
Session.MarkedOnly(true);
end;
Fine-Tuning the Threshold
The MinimumCosineSimilarity value in the SemanticSearch procedure determines how strict the search results will be. A higher value will return fewer but more relevant results, while a lower value might include more sessions with varying degrees of relevance. Experiment with different thresholds to find the right balance for your copilot. On average, a threshold of 0.7 to 0.8 is a good starting point.
Debugging and Validation¶
Test the semantic search implementation by interacting with the Event Assistant Copilot and observing the results. If you are using the sample Sessions Excel file, try asking something like:
When is the session about sales ?
and you should get info about sales and marketing sessions.
Conclusion¶
Semantic search is a powerful tool that enhances the user experience by providing more relevant and context-aware search results. By implementing this feature in the Advanced Assistant Copilot, you can improve the accuracy and usability of your AI-powered assistant, making it more intuitive and user-friendly.
Congratulations!¶
You've successfully implemented your Advansed Event Assistant Copilot with semantic search capabilities. Your copilot can now understand the context and meaning behind user queries, delivering more accurate and relevant search results. Great job! 🎉