Large Language Model in Creative Work: The Role of Collaboration Modality and User Expertise

Since the launch of ChatGPT in Dec 2022, Large Language Models (LLMs) are rapidly adopted by businesses to assist users in a wide range of open-ended tasks, including those that require creativity. While the versatility of LLM has unlocked new ways of human-AI collaboration, it remains uncertain whether LLMs can truly enhance business outcomes. To examine the effects of human-LLM collaboration on business outcomes, we conducted an experiment where we tasked expert and non-expert users to write an ad copy with and without the assistance of LLMs. Here, we investigate and compare two ways of working with LLMs: (1) using LLMs as "ghostwriters," which assume the main role of content generation task and (2) using LLMs as "sounding boards," to provide feedback on human-created content. We measure the quality of the ads using the number of clicks generated by the created ads on major social media platforms. Our results show that different collaboration modalities can result in very different outcomes for different user types. Using LLMs as sounding boards enhances the quality of the resultant ad copies, especially for non-experts. However, using LLMs as ghostwriters did not provide significant benefits and is in fact detrimental to expert users. We rely on textual analyses to understand the mechanisms and learned that using LLMs as ghostwriters produces an anchoring effect which leads to lower-quality ads. On the other hand, using LLMs as sounding boards helped non-experts achieve ad content with low semantic divergence to content produced by experts, thereby closing the gap between the two types of users.