The Genesis of Insight Generation

Professor John M. McCann
Fuqua School of Business
Duke University

Insight generation was conceived in the early days of the Marketing Workbench Laboratory, where the focus was on building prototype expert systems for use in consumer packaged goods sales and marketing. As with any invention process, we went down a number of different paths during the 1986-1990 period. These paths involved many different expert system technologies and a multitude of marketing and sales problems. The research approach was experiential in nature: build an innovative system which solved a marketing or sales problem and learn new things via the process of building the system and presenting it to sales, marketing, and systems managers in a large number of companies. In the beginning, there were no clear visions of the underlying theme of the various innovations, other than the use of expert system technologies for producing the systems. The basic idea was to the find a problem, solve the problem, use introspection to arrive at the methodology being used to solve the problem, and then use an expert system shell to build a expert system which solved the problem.

Once a number of the systems had been built, we did a form of meta-analysis to determine the features shared by the disparate systems. This analysis led to a recognition that the early systems were all writing short reports, in English, which summarized the findings of the application of the expertise in the system. We could see that our operative model of the user of the systems was a reader ... someone who reads reports rather than analyze data.

This recognition led us to overtly apply a "writing model" that evolved over time through discussions and statements that are reflected in the following:

• If only the system were as easy to use as my word processor.
• If only I could simply compose a report in my word processor, and then have an expert system come along behind me and correct my work. If I were to write sales are up 4% this quarter, I would like the system to check this fact and make corrections, if necessary. For instance, it might look up the correct sales figures in a database and then replace my sentence with sales are down 1.3% this quarter.
• OK, I give up. I realize that we cannot write a system which will check the facts underlying our writing as we write it. But perhaps it does not have to be so smart. I bet that most (80-90%) of the things which appear in most marketing reports are very similar to paragraphs that have appeared in other marketing reports. There are only so many things that one would want to say. Let's try to make a system where one would write a new report by pulling together segments of older reports. Once the user selected an old segment, the system would re-write it for the user; for his/her brand and time period.
• Hey, this is great. I can compose a new report from segments of old reports.
• Although I am writing a report, I am actually doing analysis at the same time. I tell the system what paragraph to write, it does the required analysis and writes the report by re-writing old material in light of the new analysis. Let's call this approach analysis by composition to denote the fact that we are doing analysis via the process of composing a report.
• It looks like we will need many small programs which do a specific piece of analysis and then write a paragraph about its findings. Let's call these programs analyzers.
• Once we have written a report, we know something new about the brand and/or market. Wouldn't it be nice if the system could monitor this new knowledge and let us know when it was no longer true. Hey, we need monitors.
• OK, we have our system running and solving a problem we invented. Let's ask our sponsoring companies to give use some real-world problems. We can then see if our analysis by composition approach is useful in writing prototypes which solve their problems.
• Do you know what we are doing here? We are building systems that generate insights; let's call them insight generation systems .
• We have been generating these insights "on demand" by allowing the user to determine when new insights and reports are generated. What happens if we put an agent in a loop and run it for all items in all markets?
• I saw where there is a company using the phrase active documents . That sort of applies to our work. But I like the phrase dynamic documents. And their documents are not that data oriented, as are ours. How about dynamic data documentsto denote the fact that the document changes over time in response to data.
• Each time we run an agent, we get new insights. And, a new report is generated. Pretty soon, we are going to be inundated with insights and reports. We need to be able to "know what we know." We need an insight management system.
• Our analyzers and monitors are like Alan Kay's agents: soft robots that live in the computer and look for things that they know the user would be interested in. Let's call them agents. We will have a lot of agents. What should we call the collection? Some guys at Knowledge Systems Corporation have written a paper about agents and agency. Let's see if their ideas are useful in our framework.
• I'm going to Japan to present these concepts and applications to several companies. We need a bigger concept around which we can organize all this stuff. How about the notion of a Marketing Factory?

These thoughts and snatches of conversations occurred over a four year period. One can see how new ideas flowed from a yearning to make the word processor the center of the analysis process and involved the progressive/sequential process of building a prototype system and then thinking broadly about it to find the next step. This hyperbook takes the reader along the journey we traveled. It presents our ideas, concepts, and prototypes so that the reader can come away with a clear understanding of an evolutionary path for marketing systems.