Edited by Peter Steidl, Behavior change expert

Technological progress is disrupting industry and commerce, making old ways of working and legacy products redundant. Will it have a material impact on market research practices? Will AI enhance research services or disintermediate market research?

Here are some of the way advanced technologies, and in particular AI, can enhance market research practices.
AI can –

These are by no means all the contributions AI can deliver, but this selection is painting he picture: market research will become more reliable, provide results faster and at lower cost, change work practices, and make some positions redundant. Unlike you fall into the latter category you will be able to enjoy a revitalization of market research methodologies and practices.

There is a risk, however: as AI tends to make everything easy and effortless, it will encourage some users of research results, who lack domain expertise and understanding, to undertake their own studies. Lacking a basic understanding of research methodologies, they may well come to misleading conclusions.

But AI will not only revitalize market research practices while causing the redundancy of some staff, but it will also disintermediate market research in some applications. More specifically, there are two developments that should give research firms pause:

The use of synthetic data has intensified, and new applications are coming on-stream with increased frequency. Essentially, they are creating a digital twin of a target group based on research and, where available, other data, allowing the user to interrogate the synthetic data rather than undertaking an additional survey.

There is no doubt that synthetic data can be useful, reducing the cost and time required to get important answers, while also allowing for testing a multitude of options – a process that would be hugely expensive and time consuming when using market research.

The use of synthetic data is not new. Digital twins of manufacturing, warehousing, supply chain and other environments have been created for some years now, allowing for the optimization of these operations by testing various options using the digital twin. When changes in the real world are made, it is possible to update the digital twin to ensure it does represent the real-world entity and processes.

Digital twins can also be useful in marketing applications. For example, we have created a digital twin of a major supermarket to test how traffic flows can be shaped, and primes utilized in the most effective way. The point is that digital twins can be extremely useful if they represent the real world we want to investigate.

Of course, with operations such as manufacturing or warehousing we can update the digital twin whenever we make changes in the real world to ensure the digital model remains a twin. The supermarket application I mentioned, on the other hand, was used to support immediate action, i.e., the results the digital twin delivered were relevant as they reflected the environment, we wanted to change at the time decisions were made.

However, there are indications that some marketers and their agency use synthetic data in applications where the results are likely to be misleading because the digital twin is not any longer a true representation of the real world. Let me explain.

Imagine synthetic data based on market research results, which may have been generated using direct questioning, applications such as eye tracking, or observations. This data represents our target group – it is our digital twin, and we can interrogate it using AI.

This raises several questions:

Does the (aspect of the) real-world target group we have replicated change over time, or is it likely to remain static?

The answer to this question is sometimes related to the planning horizon. For example, when I create a digital twin of voters for an upcoming election, it is unlikely that they change their attitudes, opinions, beliefs, and intents dramatically unless there is a disruptive event – and the latter is a rare occasion. This ensures that my synthetic data is mirroring the real-world target group of voters, and I can interrogate it to get answers.

But what happens when there is a relevant real-world change that impacts on how consumers feel about your company, brand, or product?

A competitor or a breakthrough innovation (e.g., electric vehicles) is disrupting the market or you are planning to launch a disruptive product or campaign. In these situations, it is unlikely that an interrogation of synthetic data will deliver reliable results. The data your twin is based on comes from the time before a disruptive event changed the market. Similarly, when you interrogate your digital twin with respect to a disruptive product or campaign, it cannot provide reliable answers as consumers need to see it to provide informed responses.

Similarly, when there is a radical change internally, say you must reduce the workforce or take the organization through a massive transformation, you may want to know if your actions will impact, or have impacted, on how employees feel about your company and their work. Again, the synthetic data won’t help, as this is a new scenario, and historical data is likely to be misleading.

Of course, if we can update the synthetic database regularly, your digital twin would still represent the real world. But to update it would require a survey of our target group and this is exactly what the digital twin is supposed to allow you to avoid.

Finally, we must question if synthetic data based on a generic data pool is likely to deliver insights that are relevant to your target group. For example, synthetic data is being used to assess advertising concepts, images, headlines and more. Can we assume that all people will react in the same way to these stimuli? Would, for example, a dog owner looks for different aspects on dog food packaging compared to somebody who has no emotional relationship with a dog?

In my experience users of such synthetic data rarely ask how it was generated, if it is being updated regularly, and if it is possible to select data representative of the target group for an assessment.

In summary, there are some critical questions that should be considered before the use of synthetic data:

These are broad guidelines. Here are some specific questions you may want to ask:

If you are satisfied that the synthetic data does, in fact, represent your target group like a digital twin, you will be able to save significant expenses by interrogating it, rather than conducting primary research. You just must make sure that it does reflect your target group with respect to all relevant aspects.

In conclusion, we see digital twins replacing primary market research in several applications. But at the same time, marketers and their agencies need to be vigilant and ask the right questions to ensure that the use of synthetic data is appropriate and likely to deliver reliable results.

Let’s say you want to create communications – images, headlines, copy – that shapes the purchasing behavior of loyalty members exploring our offer online. In the past, you might have undertaken a survey of this target group to test various concepts, allowing you to select the most effective option.

But AI allows you to take a different approach: You create an inventory of concepts. AI assesses the characteristics of each individual loyalty member based on personal and purchase data available and selects for each person what it deems to be the concept with the greatest impact.

Over time, AI gets better and better at selecting the most effective concepts as it learns from every application, knowing if the member has bought or not and, if so, what has been bought and how this differs from past purchases. Depending on what your organization offers, AI may also be able to identify indications that a particular interest in a product category is developing. For example, a shopper exploring or buying some baby related items may well be on the way to become a parent, suggesting certain offers may be relevant and encourage purchases. Or a person looking at camping great may plan a tracking holiday, and so forth.

I have just outlined the perfect application opportunity. We know a lot about loyalty members and therefore there is much data AI can use to select the most effective communications concept. Unfortunately, many potential applications suffer from limitations:

First, we may not know much about the individuals we want to influence. In some applications AI can still make use of some generic factors such as the weather, time of day, weekday, etc. to infer a certain state of mind or mood, and their behavior may also allow us to establish if they are exploring or focusing on a specific purchase, product category or application.

Second, AI may have difficulty identifying specific individuals. For example, in a store environment there may be several shoppers clustered together. While sensors may be able to identify individuals and some of their characteristics, the display of relevant material may be hampered by having several shoppers exposed to the same messages, when this message is perfect for just one of them.

These are considerations we need to entertain to decide if real-time optimization is an appropriate option. The question is not ‘is this a perfect opportunity’ but rather ‘is real-time selection of exposures likely to have more impact than current practice’. For example, serving the right concept to the right party with online media may not be always perfect, but nevertheless may allow for a degree of personalization that is likely to boost the effectiveness of your communications.

From the market research perspective, we note that the use of AI to select and serve concepts, while learning from their impact and optimizing the selection over time is replacing market research studies a-priory testing concept, selecting the most effective one to be used across a large number of consumers. The latter approach restricts the range of options that can be tested, while the use of AI optimization can accommodate an almost unlimitednumber. Importantly, the use of AI allows for ongoing optimization taking responses into account, while the market research approach offers data that will eventually lose its representativeness with time passing.

In summary, AI can have a material impact on market research practices and disintermediate some types of studies. There is little doubt that it will change the market research industry.

The second part of this article is available here.