Skoda X ran a validation POC from January to March 2024.
We designed a custom model to replicate Skoda Auto’s customer base.
We validated accuracy of the model with archive and ongoing research projects with 36 testers.
Pilot involved 12 stakeholders from sales, strategy, and CX to gather views on AI-powered surveys and build awareness.
Company overview
Skoda X (SX) is an innovation vehicle of Skoda Auto. This department scouts for cutting-edge solutions that could help optimize various tasks, can help build new competencies for SX or create new services at Skoda Auto and potentially the whole Volkswagen Group.
Time
Q1 2024
Timeline
January = Bootcamp
February = Labs
March = Testing and trying
May = Validation
June = Organization
July = Pilot roll-out
August = Large-scale validation
Main goal
Full customer surveys using AI in a pilot mode based on topics from Skoda Auto's sales and marketing department. Then comparing them with parallel surveys on the same topic with classic (human) respondents.
To check whether SX can do research projects in-house with AI and business analysis of using AI for customer researches (benefits, costs, risks, process...)
Pilot topic
E-mobility
People involved
Key stakeholders: 2
Active users: 14
WOW at demo: 20
Departments: Innovation, Strategy, CX
Roles: product owner, business owner, research manager, innovation manager
Who participated
Heroes of the project
1
Research costs too much
Skoda does tons of research and testing. Can we optimize costs for customer researches with AI? Can we deliver all customer researches?
With Lakmoos, they aim to save the budget while increasing (not decreasing!) the amount of collecting opinions.
2
Secret features
Skoda needs to test without spreading the word that something big is coming up.
Lakmoos models cannot leak information about new products.
3
Research takes ages
Approving a new research project can take months, but Skoda needs answers today. Before they analyse the answers, the market has changed.
With Lakmoos, employees can bring answers to their management in time. In fact, they get instant answers.
Slide from internal presentation of innovation InnoDay 2024 in Mladá Boleslav: The problems with research that it takes ages, costs a fortune and yet is used only once.
4
Hard-to-reach groups
Most testing is done on their clients for convenience.
With Lakmoos, Skoda can easily ask the same questions to Kia or Hyundai owners.
During the pilot, we understood how Lakmoos works, understood the benefits and limits of the Lakmoos model and modelled the business case for the upcoming years.
Tomáš, Innovation
What we delivered
Results
Custom model that reflects customer groups and safely integrates archive research
Testing with 36 internal testers from Skoda Auto with > 1345 questions asked
5 full research projects, from brief to report
Full training for a department to in-source (as opposed to out-source) research projects
The best questions in Interviews
Have you ever considered purchasing an electric vehicle?
What would motivate you to buy an electric vehicle?
What electric range would convince you to buy an electric car?
Example of a Likert-scale answer in automotive.
Survey brief example: Online car storage
We want to gather insights about user preferences in their user journey through ŠKODA website. We want to see users’ attitudes towards registering on websites, how they like to save preferred products, and their opinions on using product watchdog or app notifications during the purchasing process. Past research has shown that users prefer simple registration process, relevant notifications, and that they prefer heart over star as a button to save preferred products. We want to see whether these findings still hold after two years.
After consulting with SX team, we created strategic personas that represent wider groups of their target customers. Besides demographic info, we filtered these people by lifestyle and banking behaviour, e.g. how much time they spend in the nature.
How do we change the game?
Each answer is quantified. In both modes, Qualitative interviews and Quantitative surveys, we always compute how many people agree with statements. This way, all open questions have a scale for comparison: Does this answer represent 10% or 90% of our customers?
Instant answers. The system is fast. In Qualitative Interviews with average representant of a wider group, answers are computed instantly. In large Qualitative Surveys with hundreds to thousands of respondents, results take minutes to hours.
Flexible target groups. We configured default respondents based on different niche groups with no demographic boundaries. This way, we also modelled haters of e-mobility, competitors’ clients or hockey fans.
A slide from a POC evaluation presentation with benefits of Lakmoos compared to human samples: quantifies answers, instant answers, custom target groups.
January to march
POC
The goal of the Proof of Concept (POC) is to understand the functionality, possibilities, and risks of Lakmoos models and to explore their use for the needs of Škoda X and Škoda Auto. For this purpose, Lakmoos will train a model on the topic of e-mobility in the Czech Republic. In addition to jointly testing the model (Lab), we will also plan workshops and lectures that will contribute to a better understanding of the system. A secondary objective is to understand (saved) costs and quantify this new method of research.
Five expert workshops
Hundreds of questions asked and tested
Complex evaluation of usability of Lakmoos in future projects
POC competency matrix. Each competence was linked to a project phase, week and type of activity (workshop, talk, lab, experiment).
May to august
Validation and ramp-up
After initial validation of the model during POC, our next steps were to:
Validate model with existing research projects
Distribute a limited number of licenses to different departments in a controlled manner
Move beyond Skoda Auto to the Group level
Most of the metaphors you will hear from us were born in the labs with Skoda X. This is one of many: LLM is only a shell that knows how to look human. With good prompting, it can lift a hand and wave. We create brains to make it alive and act with purpose: wave at a friend.
It is so refreshing to work with a young and hungry team. You always have an update that boosts the product and the speed is just impossible.
David
For example, our model accurately simulated daily mileage. There is not data about the mileage but other datapoints (from the hudreds thousands datapoints) got to the same conclusion. Simply put: When the model know where a person lives, where they work and how many times per month they go for a trip, it can predict the mileage accurately.
Jan, CTO at Lakmoos
First reality check in Qualitative Interview
Drop-checking was done already at the workshops with people who could access PowerBI or had datasets for comparison. For instance, we checked our daily mileage with real data this way.
A 20-page report compared results from AI research and traditional research.
Comparing Quantitative Surveys with a previous research project
The two goal of the validation were to
Showcase the future service of the in-house expert team providing AI research
Discover similarities and differenes in results, process and the effect on time and cost of completing a research project
We compared results to 10 quantitative hypotheses (= if and how many people) and 3 qualitative hypotheses (= why).
Outcome: Yes, Lakmoos model can answer in the same or higher quality.
Higher quality was found in
Always adding % (Group fit), even to open questions and qualitative questions asking about worries and motivations.
Lakmoos can handle design questions, albeit not visual.
For full transparency, we also outlined the advantages of human samples
More suitable for in-depth feedback for design questions (with visual input).
It is easier to trust humans.
However, this comes at the cost of poor generalisability to the target population from the sample.
Is the system safe?
Yes, we met with Compliance and Data security departments in ŠX in the POC.
Together, we prioritised data sourced to integrate into the custom model in various phases.
How accurate will the answers be?
Validation comes at three phases: POC, Validation and Roll-out.
In short, answers from Lakmoos are just as accurate as real people BUT they are 1. always quantified, 2. fast and 3. unlimited.
Counting in all the costs, are Lakmoos surveys still faster and cheaper?
We were also curious about the process. In theory, answers are instant, but how long does it take from brief to report? How much time do internal employees spend with Lakmoos compared to briefing an external agency or collecting data themselves?
We ran an internal test to see that doing full-scale research with Lakmoos takes less than two weeks, brief to presentation.
Lakmoos is faster and cheaper.