How LLMs May Change the Publishing Industry
— LLM — 2 min read
Basic Facts:
- People use LLMs for paper writing.
- It is difficult to determine if LLMs were used during the writing process.
- Writing issues impact paper acceptance, and LLM usage can mitigate them.
From these facts, it is clear that using LLMs in paper writing is inevitable (as a comment to [1]). However, how can we effectively utilize them for the advancement of science?
LLMs play two roles in the publishing industry: author and reviewer. From the writer's perspective, LLMs can be used to refine the paper by improving wording and identifying unclear points. The question of whether LLMs can generate truly innovative ideas, or "novelty," is interesting. Human reviewers often consider lack of novelty a weakness when they find the paper uninteresting or similar to previous work. So, what defines novelty and the process of generating genuinely novel ideas? Let's approach this from a machine learning perspective. Aha moments occur when we discover something correct but previously unthought-of. Therefore, novelty could be defined as sampling solutions that lie outside the established distribution and lead to the correct result. This distribution is typically constrained to a specific area, as analogical arguments are not considered a lack of novelty (?).
Regarding reviews, the challenge lies in improving human-AI collaboration for reviewing. It is important to consider the cognitive limitations of humans and explore how AI can assist them. While Liang et al. [2] demonstrated that GPT-generated reviews can be helpful, separating human and machine reviewers may not be ideal. For instance, machines may struggle to provide in-depth critique of method design, while humans may overlook specific details that machines, using strategies like RAG, can handle effectively. A possible review process could involve human reviewers posing questions, and LLMs assuming the role of the author to respond based on the paper. This iterative process allows the author to refine the paper until it can no longer provide satisfactory answers. Additionally, summarization and improved interaction with the paper can alleviate the burden of reviewing.
To conclude, here are some potential directions for research and writing:
- Develop better strategies for human-AI interaction in writing reviews.
- Define the informativeness and novelty of papers and reviews. To determine the former, all existing papers may need to be considered.
References: