Why AI chatbots depend on high quality data: Garbage in, garbage out explained

Chatbots and digital assistants are one of the emerging trends in the digital banking industry. According to Juniper research, over 80% of banks are now using some form of chatbot in customer service. Lately, more and more banks are also starting to shift to more analytical usage, such as personal assistant, which reshapes the user experience, offering personalized financial advice and assistance at the touch of a button. But every good analysis needs a well structured and organised data first.

1. Chatbots and how they can help

Many types of AI chatbots help bank users achieve their financial goals. From customer service to financial advisors and guides, their core functionality works quite similarly, using the data to understand the situation. Here are a few examples:

‍Finn is a chatbot based on the GenAI platform from Dutch neobank bunq that gives users instant insights on their spending, budget, or places visited. Users can ask questions or seek advice about their bank account, savings and anything else related to their finances.

Trim is an AI-powered personal finance assistant designed to help users save money effortlessly. It identifies potential savings opportunities through bill negotiation and subscription cancellations. Automates savings transfers to help users build financial resilience and achieve their goals.

‍

Cleo combines AI technology with a conversational interface to offer personalized financial insights and guidance. It offers personalized financial insights and recommendations based on user transaction data. Simplifies budget tracking, goal setting, and expense categorization for improved financial management.

‍

2. Garbage in - garbage out

At the heart of any AI-powered system lies a simple concept: Garbage in, garbage out. In other words, the quality of the output is directly dependent on the quality of the input data. This principle underscores the importance of data cleansing and enrichment in ensuring the effectiveness of AI advisors and chatbots.

Naturaly this can work also through platforms like ChatGPT from OpenAI, so we decided to dive deeper with an actual test to see how much a good enriched data effect the results.  

‍

3. Testing ChatGPT

Let’s see what happens if the advisor works with richer and more accurate data vs. raw unstructured information.  

To illustrate the impact of enriched data on AI financial advisors, we used a data analyzer of the newest ChatGPT 4.0 platform and our own payment history. Armed with identical questions, we asked for analysis and financial advice with raw payment data first, including information like the merchant identifier, simple description, city and country location or MCC codes. We then repeated the process with the same data enriched by TapiX API, which gives data meaningful structure. That includes information like GPS location, logo, exact address, Google Place ID and detailed category tags. Chatbot was not improved in any way, nor trained for this type of questions.

The ultimate goal is to have a meaningful discussion with a chatbot about finances which covers data interpretation and basic insights, recognition of spending habits, and ideally getting some help with setting up and reaching financial goals. Therefore the test is divided into 3 segments – categorisation, spending habits and financial goals.

‍

‍

‍

Experiment #1: Transaction Categorisation

First, we asked the simple questions - What are the top 5 merchants with most spending? What are our favourite restaurants in the food category? Can it create a pie chart with the most spending categories for better understanding?

Non-Enriched Data

As expected, usage of non-enriched data causes misclassification of transactions, leading to more misunderstandings and financial strain for the user. In this case, ChatGPT did not present the top 5 merchants correctly, which not only looks untrustworthy but can also lead to a wrong financial decision later on.  This also shows that non-enriched data are unsuitable for any PFM platform due to a lack of proper details.

‍

Enriched Data

ChatGPT equipped with enriched data was able to categorise our annual payment history accurately. After summarizing the spending amounts into each category, we asked to create a pie chart to represent the distribution of spending in different merchant categories. ChatGPT was also able to create a geographical spending map to help us understand where the most expenses are coming from and where the user can adjust the spending.

‍

Experiment #2: Spending Behavior Analysis

We continued with deeper questions – What are some areas where we spend the most money? Can chatbot give us tips to save money in each category and adjust our spending? (SEE DISCUSSION IN PICTURE 1)

Non-Enriched Data‍

Non-enriched data allowed the chatbot to base its tips on regular MCC codes which he can read and understand. This was already a great non-trivial achievement of LLM. However, without proper contextual data or additional information from TapiX API, the resulting advice is very marginal. The chatbot simply doesn't have enough data to know what it is looking at.

‍

Enriched Data

With enriched data, ChatGPT was able to segment spending behavior into key areas like travel, accommodation, home goods, transportation or dining out, giving several detailed suggestions in each category. Tips were more focused on personal spending behaviour that ChatGPT learned from proper categorisation.

‍

Experiment #3: Financial Goal Understanding

Non-Enriched Data

‍Without enrichment, ChatGPT provides generic savings goals based solely on income percentages, failing to address specific user aspirations. Inaccurate categorisation also makes it hard for chatbot to correctly read and understand specific payments, like subsriptions, negatively affecting its possibilities as financial advisor.

‍

Enriched Data‍

ChatGPT equipped with enriched data could accurately understand users' financial goals with current spending patterns, and offer customized tips to save money. In this case, ChatGPT was able to analyze recurring payments and identify potential subscriptions thanks to the information in the category tags, including a number of transactions for each, and give tips for potential subscriptions worth cancelling.

‍

How to try this ChatGPT experiment with your own data?

If you want to try a similar experiment, we give you a complete communication with ChatGPT chatbot (with enriched and unenriched input data) and a sample CSV input document.
‍

Full ChatGPT communication with input from enriched data.
View here

Full ChatGPT communication with input from non-enriched data.
View Here [Part 1], [Part 2]

Download CSV sample data structure.
‍Download here

‍

Conclusion

As you can see, LLMs can do great work with analyzing data but it is still not enough to overcome problems caused by incomplete and information-poor inputs. When working with non-enriched data, results were mostly incorrect and advice meaningful and applicable, but very generic. Such results will only undermine the bank’s credibility.  

Taking good care of your core data management not only enhances the performance of AI advisors but also fosters trust and loyalty among users. By leveraging enriched data, banks can deliver hyper-personalized experiences that resonate with users, driving engagement and ultimately, bolstering their competitive advantage in the digital banking sector.

In short, AI chatbots can be a real “quick win” solution, but only if high quality data are there in the first place.

‍

‍