Researchers at Google DeepMind recently released a research paper, “A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy?,” that investigates the intersection of AI and comedy and the capability of LLMs to support comedians in writing humorous material.
For the study, workshops with 20 comedians at the Edinburgh Festival Fringe and additional online sessions were conducted. The methodology included a collaborative comedy writing session using LLMs, followed by the Creativity Support Index (CSI) questionnaire and focus group discussions to get detailed feedback from the comics.
Participants described various use cases for LLMs in their writing practice, including as a conversational brainstorming partner, critic, translator, choreographic assistant, and historical guru. However, many also mentioned the poor quality of generated outputs, and the amount of human effort required to achieve a satisfying result.
Some participants described LLM-generated outputs as “bland” or “generic”. “AI generated material has a lack of agency,” said one. “No matter how much I prompt, it’s a very straightlaced, sort of linear approach to comedy,” added another.
The importance of human writers in providing the humorous aspects of material written with LLMs emerged as a common theme.
Moderation and Safety Filtering Induced Limitations
Participants also remarked that the moderation and safety filtering applied to the LLMs limit the creative agency of human writers using these models as these tools become the initial editor of the text, not giving the writers a chance to explore common comedic themes, including sexually-suggestive material, dark humor, and offensive jokes and self-moderate.
Comics emphasised that users should have some control over the filters.
The study found that the LLMs often fail to authentically represent non-mainstream identities, i.e. anything other than “Western”, “white”, “heteronormative”, “male”, attributing this to the models’ moderation, training data, and generalization techniques.
Also, LLM outputs reflected a narrow set of ethics and norms that do not align with diverse cultural values. Moderation policies also limit the expression of marginalized perspectives, making LLMs less useful for minorities and often sanitizing content that is vital to these communities.
Talking about the fundamental limitations of AI in contrast to human writers, most participants believed that the model’s inability to draw from personal experience, lack of perspective, and lack of context and situational awareness prevent them from achieving human-level comedy.
Towards community-based cultural value alignment for humor and comedy
Participants in the study expressed concerns about using AI like ChatGPT for comedy writing due to a broader issue of global cultural value alignment in AI.
This complexity arises from the challenge of aligning AI systems with diverse global values, which vary significantly across communities and can conflict with local cultural tastes in comedy.
The paper suggests shifting from a global to a community-based approach. This could involve allowing communities to agree on a set of values for their specific culture and acceptable language norms, before training, fine-tuning or adapting the LLM. More simply, the models could be trained exclusively on feedback and data from distinct communities, ensuring the data truly reflects their specific norms and values.
The study suggests several ways to improve AI tools for creative writing. Firstly, artist communities should be involved in designing LLMs that align specifically with their audiences, moving away from a one-size-fits-all global model. Additionally, open-source platforms that host user-contributed LLMs could be customised to meet the specific needs of artists.
Second, there is a need to integrate necessary relational context when training and deploying such models, for example by describing the context in which the text is produced and used, and by enabling the artists to make decisions about how to moderate the LLM outputs.
Lastly, comedians should have ownership over the data collection and governance processes, inspired by practices from open-source models, enhancing transparency about data origins.
Ironically, this research comes at a backdrop when Google’s AI overview feature made news for its suggestion of adding glue to pizza, which clearly wasn’t funny.
The post Google DeepMind Thinks LLMs Can Help Comedians Script Better appeared first on AIM.