EXPRESS: AI and Its Impact on Creativity and Diversity: An Empirical Study of LLM-Generated Product Ideas

This research examines how well large language models (LLMs) generate new product ideas, focusing on new products for college students priced under $50. Through a series of studies, we identify key strengths and weaknesses of using LLMs for product innovation. Our first study shows that LLM-generated product ideas have higher average quality than human ideas (based on purchase intent) and are 7 times more likely to rank in the top 10%. Our second study demonstrates that this AI-induced creativity boost is not explained by the LLM’s more persuasive pitching skills. Our third and fourth studies identify an interesting weakness of using LLMs for brainstorming, showing that AI-generated ideas are less novel (at the idea level) and less diverse (at the set level). In our fifth study, we investigate the generalizability of these findings by analyzing prior LLM-based creativity studies, finding consistently lower idea diversity across all of them. Our sixth and seventh studies investigate the efficacy of various techniques to mitigate this diversity loss. Specifically, we compare different LLMs (different vendors and different versions) and find that the more recent models are capable of generating more diverse ideas though still falling short of achieving human-level diversity. In addition, we demonstrate the effectiveness of several techniques that can be used to increase idea diversity almost to the level of human idea generation: pooling ideas across vendors, prompt engineering (including Chain-of-Thought prompting and injecting heterogeneous personas or constraints) and creative agents that broadly explore the solution landscape with the intent of restoring diversity. Finally, in our eighth study we show that exploiting the near-zero marginal cost of AI idea generation by scaling the number of ideas steadily improves coverage of the idea space approaching human-level coverage. We conclude our work by presenting a set of actionable recommendations targeted to managers of innovation that want to identify better new product ideas with the help of LLMs.