What was the context or situation that gave rise to the need for the project?
The client MCH Group, operates several physical events in Switzerland and each event has a website that promotes, informs, and helps users register for those events. Each event hosts several exhibitors who will interact with many visitors over several days. Each event focuses on a different domain and Industry, ranging from Gardening to the Swiss Electricity industry. (e.g. Powertage).
They were looking to have a virtual assistant on their event websites so that visitors and exhibitors alike can ask questions and get answers quickly, without having search through multiple pages. Having had prior experience with other virtual agents providers ranging from costly solutions which cost 99 cents per answer/session to ones with a very low accuracy, they wanted something better and more cost-effective. Together with our partner Eclipsai we agreed on a set of features for the CWYD (Chat with your data) solution.
What were the key challenges or problems that the project aimed to address?
Web scraping the content from the website and chunking/structuring the content in such a way that is optimal for retrieval.
Masking PII information when storing the conversation history to be GDPR compliant – using AI Text Analitics to identify PII entities and replace them with **** when stored.
Getting to 80% precision, according to a set of 40 Golden Truth pairs of Questions and Answers provided by the client (most commonly asked questions expected)
Answering questions which required more than 5-10 chunks (content which is scattered throughout the website, like the list of all exhibitors which might have their own dedicated page and would end up chunking 50+ different pages.)
What were the specific goals or objectives that the project aimed to achieve?
Create a virtual agent that is easy to use, fast, accurate and cost effective.
How were these objectives measured or evaluated?
We have an analytics Power BI report that measures what was asked, the feedback from user (thumbs up/down) for each answer.
We measure accuracy/precision by using an automated evaluation framework that issues questions to an LLM, gets the answer, compares the answer to the golden truth answer (for certain metrics) and gives a score from 1 to 5 (1 worst, 5 best)
What was the main solution or approach that was developed or implemented?
Our virtual assistant is powered by the latest features in Retrieval Augmented Generation (RAG), utilizing state-of-the-art tools like Azure Cognitive Services and Azure Open AI. The modular architecture allows for flexibility and cost-effectiveness, enabling you to swap the core LLM model as needed. Its UI is fully customizable, ensuring a seamless integration with your event websites’ look and feel, and can be installed with a simple “one code snippet” process.
What were the key features or components of the solution?
Security and compliance are top priorities for us. Our solution is GDPR compliant, and we offer secure hosting options, either within your existing Azure infrastructure or hosted by us. Private data and conversations with the virtual agent are protected behind a Virtual Private Network and Firewall, ensuring that sensitive information remains secure.
To help you gain insights and improve the solution’s performance, we offer conversation logging for statistical purposes. This allows you to identify the most common questions and gather feedback from users on the effectiveness of their interactions with the virtual agent.
Our solution aims to achieve an accuracy or positive feedback score of at least 80%, ensuring that your event attendees and exhibitors receive satisfactory answers to their queries. By investing in a customized virtual assistant with a higher upfront fee and a lifetime license, you can avoid the ongoing costs associated with yearly subscriptions or usage-based fees from other SAAS providers.
How did the solution address the identified challenges or problems?
Web scraping was addressed using advanced frameworks which scrape web content then clean the content and discard HTML tags, menu items and other non-relevant stuff.
Masking PII using the Microsoft Azure Text Analytics framework which can accurately identify a wide range of entities.
Adjusting Chunking and Retrieval using an iterative approach, by measuring precision using the evaluation framework, adjusting the RAG pipeline and retrying until we got an accuracy of 78-79% in the end.
How did the solution address the identified challenges or problems?
Project was delivered on time and proved to be a success for the end client.
What technologies or methodologies were used in the project?
The solution was designed to work with an Azure Cloud Infrastructure, but can be adapted to work in any cloud environment, LLM can be replaced, Vector Database, Orchestrator app server etc.
The solution in action
Next step?
At mindit.io, we are ready to help the client undergo a technology-driven transformation.
Let’s shape the future together with technology-driven success!
