Rogue Thoughts

If you have purchased something on Amazon, iTunes or the like you will already have come expect that they will recommend something else, which you might like. Would that work in the supermarket? Superficially a yes seems probable, but does it take the same kind of intelligence to shop books as bananas?.

Recommendation engines are a simple form of artificial intelligence. (at least compared to the more illustrious versions known from popular culture – such as the film “Bladerunner” or “I Robot”). The term artificial intelligence was originally coined in 1956 by John McCarthy and has since then followed two branches. One branch is philosophical and ponders the nature of the human mind, while the other is more technical and tries to apply artificial intelligence in technological solutions.

The technical branch has brought us the algorithms that exist today on the web in the form of recommendation engines like Netflix, jjester, stumble upon, last.fm etc.. This particular type of algorithms are called collaborative filtering algorithms. They have been fine tuned in the extreme, but they are all aimed at one particular problem: recommending a limited number of new items that the customer/user has not yet tried from a vast repository. One key characteristic of these items, whether they be films (Netflix), jokes (Jester), websites (Stumble upon) or music (Last.fm) are that they are only meaningfully used once (music is a partial exception, but still bought only once).

Now, working for a food retailer, we wanted to apply the same kind of AI to retail, but found ourselves faced with the problem that the repository of possible items was not so vast, since a typical large retailer will have between 20000 and 40000 products at most. The typical consumer would buy mostly the same products repeatedly, which clearly differs from movies (I know I saw Star wars episode 5 more than thirty times, but I was young and it is an exception), jokes (although some people really do tell the same joke over and over again, that is also an exception), websites (this one is more tricky, but when you want stumble upon to recommend a website, it is a new website you haven’t used before), and music (again I must admit that I have bought Aphex Twin’s “Selected ambient works Vol. 2″ five times, but it was because it kept getting lost for some reason).

Of course, you can use the collaborative filtering algorithms to introduce a customer to new products, but often the customer may be more interested in locating offers on stuff he usually buys, or just stuff that he ran out of and has to replenish. Another thing would be seasonal recommendations, like offers on fresh berries when they are in season. I still haven’t found the optimal solution for this type of problem where you want recommendations for items that are used repeatedly and selected from a limited base where novelty is not the key driver of customer appreciation.

Obviously some sort of time dimension has to be woked in to it. There are at least two sorts of time here, the cycles that drive humans, such as the day, the working week, the year and the life cycle. But there is also the product life cycle; how often the customer needs to refill his olive oil stash and coffee stock.

I believe the solution points to a more fundamental challenge for the AI powered marketing of the web 2.0 and beyond: introducing the cyclical time dimension that drives every cell in our body, that is the diurnal, and the more cultural ones like the working week/year. That is the next big challenge for artificial intelligence of web 3.0 and beyond.