2024-11-06

Title: IdeaBench: Benchmarking Large Language Models for Research Idea Generation

• Authors: Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, Aidong Zhang
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2411.02429
• Pdf URL: https://arxiv.org/pdf/2411.02429
• Copy Paste: [[2411.02429]] IdeaBench: Benchmarking Large Language Models for Research Idea Generation(https://arxiv.org/abs/2411.02429)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have transformed how people interact with artificial intelligence (AI) systems, achieving state-of-the-art results in various tasks, including scientific discovery and hypothesis generation. However, the lack of a comprehensive and systematic evaluation framework for generating research ideas using LLMs poses a significant obstacle to understanding and assessing their generative capabilities in scientific discovery. To address this gap, we propose IdeaBench, a benchmark system that includes a comprehensive dataset and an evaluation framework for standardizing the assessment of research idea generation using LLMs. Our dataset comprises titles and abstracts from a diverse range of influential papers, along with their referenced works. To emulate the human process of generating research ideas, we profile LLMs as domain-specific researchers and ground them in the same context considered by human researchers. This maximizes the utilization of the LLMs' parametric knowledge to dynamically generate new research ideas. We also introduce an evaluation framework for assessing the quality of generated research ideas. Our evaluation framework is a two-stage process: first, using GPT-4o to rank ideas based on user-specified quality indicators such as novelty and feasibility, enabling scalable personalization; and second, calculating relative ranking based "Insight Score" to quantify the chosen quality indicator. The proposed benchmark system will be a valuable asset for the community to measure and compare different LLMs, ultimately advancing the automation of the scientific discovery process.
• 摘要：大型语言模型 (LLM) 已经改变了人们与人工智能 (AI) 系统交互的方式，在各种任务中取得了最先进的成果，包括科学发现和假设生成。然而，缺乏使用 LLM 生成研究想法的全面而系统的评估框架，这对理解和评估它们在科学发现中的生成能力构成了重大障碍。为了解决这一差距，我们提出了 IdeaBench，这是一个基准系统，包括一个全面的数据集和一个评估框架，用于标准化使用 LLM 生成研究想法的评估。我们的数据集包括来自各种有影响力的论文的标题和摘要，以及它们的参考文献。为了模拟人类产生研究想法的过程，我们将 LLM 描述为特定领域的研究人员，并将其置于人类研究人员考虑的相同背景下。这最大限度地利用了 LLM 的参数知识来动态地生成新的研究想法。我们还引入了一个评估框架来评估生成的研究想法的质量。我们的评估框架分为两个阶段：首先，使用 GPT-4o 根据用户指定的质量指标（例如新颖性和可行性）对想法进行排名，从而实现可扩展的个性化；其次，计算基于相对排名的“洞察分数”以量化所选的质量指标。提议的基准系统将成为社区衡量和比较不同 LLM 的宝贵资产，最终推动科学发现过程的自动化。

Title: Generative Emotion Cause Explanation in Multimodal Conversations

• Authors: Lin Wang, Xiaocui Yang, Shi Feng, Daling Wang, Yifei Zhang
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02430
• Pdf URL: https://arxiv.org/pdf/2411.02430
• Copy Paste: [[2411.02430]] Generative Emotion Cause Explanation in Multimodal Conversations(https://arxiv.org/abs/2411.02430)
• Keywords: language model, llm
• Abstract: Multimodal conversation, a crucial form of human communication, carries rich emotional content, making the exploration of the causes of emotions within it a research endeavor of significant importance. However, existing research on the causes of emotions typically uses clause selection methods to locate the reason utterance, without providing a detailed explanation of the emotional causes. In this paper, we propose a new task, \textbf{M}ultimodal \textbf{C}onversation \textbf{E}motion \textbf{C}ause \textbf{E}xplanation (MCECE), aiming to generate a detailed explanation of the emotional cause to the target utterance within a multimodal conversation scenario. Building upon the MELD dataset, we develop a new dataset (ECEM) that integrates video clips with detailed explanations of character emotions, facilitating an in-depth examination of the causal factors behind emotional expressions in multimodal conversations.A novel approach, FAME-Net, is further proposed, that harnesses the power of Large Language Models (LLMs) to analyze visual data and accurately interpret the emotions conveyed through facial expressions in videos. By exploiting the contagion effect of facial emotions, FAME-Net effectively captures the emotional causes of individuals engaged in conversations. Our experimental results on the newly constructed dataset show that FAME-Net significantly outperforms several excellent large language model baselines. Code and dataset are available at \url{this https URL}
• 摘要：多模态对话是人类交流的重要形式，承载着丰富的情感内容，探究其中情感的成因具有重要意义。然而，现有的情感成因研究通常使用子句选择方法来定位原因话语，而没有提供情感成因的详细解释。在本文中，我们提出了一项新任务，\textbf{M}ultimodal \textbf{C}onversation \textbf{E}motion \textbf{C}ause \textbf{E}xplanation (MCECE)，旨在在多模态对话场景中对目标话语的情感成因进行详细解释。在 MELD 数据集的基础上，我们开发了一个新数据集 (ECEM)，将视频片段与人物情绪的详细解释相结合，有助于深入研究多模态对话中情绪表达背后的因果因素。我们还提出了一种新方法 FAME-Net，利用大型语言模型 (LLM) 的强大功能来分析视觉数据并准确解释视频中通过面部表情传达的情绪。通过利用面部情绪的感染效应，FAME-Net 有效地捕捉了参与对话的个人的情绪原因。我们在新构建的数据集上的实验结果表明，FAME-Net 明显优于几个优秀的大型语言模型基线。代码和数据集可在 \url{this https URL} 获得

Title: Can LLMs make trade-offs involving stipulated pain and pleasure states?

• Authors: Geoff Keeling, Winnie Street, Martyna Stachaczyk, Daria Zakharova, Iulia M. Comsa, Anastasiya Sakovych, Isabella Logothesis, Zejia Zhang, Blaise Agüera y Arcas, Jonathan Birch
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.02432
• Pdf URL: https://arxiv.org/pdf/2411.02432
• Copy Paste: [[2411.02432]] Can LLMs make trade-offs involving stipulated pain and pleasure states?(https://arxiv.org/abs/2411.02432)
• Keywords: language model, gpt, llm
• Abstract: Pleasure and pain play an important role in human decision making by providing a common currency for resolving motivational conflicts. While Large Language Models (LLMs) can generate detailed descriptions of pleasure and pain experiences, it is an open question whether LLMs can recreate the motivational force of pleasure and pain in choice scenarios - a question which may bear on debates about LLM sentience, understood as the capacity for valenced experiential states. We probed this question using a simple game in which the stated goal is to maximise points, but where either the points-maximising option is said to incur a pain penalty or a non-points-maximising option is said to incur a pleasure reward, providing incentives to deviate from points-maximising behaviour. Varying the intensity of the pain penalties and pleasure rewards, we found that Claude 3.5 Sonnet, Command R+, GPT-4o, and GPT-4o mini each demonstrated at least one trade-off in which the majority of responses switched from points-maximisation to pain-minimisation or pleasure-maximisation after a critical threshold of stipulated pain or pleasure intensity is reached. LLaMa 3.1-405b demonstrated some graded sensitivity to stipulated pleasure rewards and pain penalties. Gemini 1.5 Pro and PaLM 2 prioritised pain-avoidance over points-maximisation regardless of intensity, while tending to prioritise points over pleasure regardless of intensity. We discuss the implications of these findings for debates about the possibility of LLM sentience.
• 摘要：快乐和痛苦在人类决策中发挥着重要作用，因为它们为解决动机冲突提供了共同的货币。虽然大型语言模型 (LLM) 可以生成快乐和痛苦体验的详细描述，但 LLM 是否可以在选择场景中重现快乐和痛苦的动机力量仍是一个悬而未决的问题 - 这个问题可能与关于 LLM 感知的争论有关，LLM 感知被理解为具有价态的体验状态的能力。我们使用一个简单的游戏来探究这个问题，其中既定目标是最大化分数，但据说最大化分数的选项会产生痛苦惩罚，或者非最大化分数的选项会产生快乐奖励，从而激励人们偏离最大化分数的行为。通过改变痛苦惩罚和快乐奖励的强度，我们发现 Claude 3.5 Sonnet、Command R+、GPT-4o 和 GPT-4o mini 都表现出了至少一种权衡，即在达到规定的痛苦或快乐强度的临界阈值后，大多数反应从积分最大化转变为痛苦最小化或快乐最大化。LLaMa 3.1-405b 对规定的快乐奖励和痛苦惩罚表现出一定的分级敏感性。Gemini 1.5 Pro 和 PaLM 2 无论强度如何，都优先考虑避免痛苦而不是积分最大化，而无论强度如何，都倾向于优先考虑积分而不是快乐。我们讨论了这些发现对 LLM 感知可能性辩论的影响。

Title: SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models

• Authors: Jianyi Zhang, Da-Cheng Juan, Cyrus Rashtchian, Chun-Sung Ferng, Heinrich Jiang, Yiran Chen
• Subjects: cs.CL, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.02433
• Pdf URL: https://arxiv.org/pdf/2411.02433
• Copy Paste: [[2411.02433]] SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models(https://arxiv.org/abs/2411.02433)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities, but their outputs can sometimes be unreliable or factually incorrect. To address this, we introduce Self Logits Evolution Decoding (SLED), a novel decoding framework that enhances the truthfulness of LLMs without relying on external knowledge bases or requiring further fine-tuning. From an optimization perspective, our SLED framework leverages the latent knowledge embedded within the LLM by contrasting the output logits from the final layer with those from early layers. It then utilizes an approximate gradient approach to enable latent knowledge to guide the self-refinement of outputs, thereby effectively improving factual accuracy. Extensive experiments have been conducted on established benchmarks across a diverse range of model families (LLaMA 2, LLaMA 3, Gemma) and scales (from 2B to 70B), including more advanced architectural configurations such as the mixture of experts (MoE). Our evaluation spans a wide variety of tasks, including multi-choice, open-generation, and adaptations to chain-of-thought reasoning tasks. The results demonstrate that SLED consistently improves factual accuracy by up to 20\% compared to existing decoding methods while maintaining natural language fluency and negligible latency overhead. Furthermore, it can be flexibly combined with other decoding methods to further enhance their performance.
• 摘要：大型语言模型 (LLM) 已展现出卓越的能力，但其输出有时可能不可靠或与事实不符。为了解决这个问题，我们引入了自逻辑进化解码 (SLED)，这是一种新颖的解码框架，可增强 LLM 的真实性，而无需依赖外部知识库或进一步微调。从优化的角度来看，我们的 SLED 框架通过将最后一层的输出逻辑与早期层的输出逻辑进行对比，利用嵌入在 LLM 中的潜在知识。然后，它利用近似梯度方法使潜在知识能够指导输出的自我改进，从而有效提高事实准确性。我们在各种模型系列（LLaMA 2、LLaMA 3、Gemma）和规模（从 2B 到 70B）的既定基准上进行了广泛的实验，包括更高级的架构配置，例如专家混合 (MoE)。我们的评估涵盖了各种各样的任务，包括多项选择、开放生成和对思路链推理任务的适应。结果表明，与现有解码方法相比，SLED 可将事实准确率提高高达 20%，同时保持自然语言流畅性，且延迟开销可忽略不计。此外，它可以灵活地与其他解码方法结合使用，以进一步提高其性能。

Title: Narrative Analysis of True Crime Podcasts With Knowledge Graph-Augmented Large Language Models

• Authors: Xinyi Leng, Jason Liang, Jack Mauro, Xu Wang, Andrea L. Bertozzi, James Chapman, Junyuan Lin, Bohan Chen, Chenchen Ye, Temple Daniel, P. Jeffrey Brantingham
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02435
• Pdf URL: https://arxiv.org/pdf/2411.02435
• Copy Paste: [[2411.02435]] Narrative Analysis of True Crime Podcasts With Knowledge Graph-Augmented Large Language Models(https://arxiv.org/abs/2411.02435)
• Keywords: language model, llm, prompt
• Abstract: Narrative data spans all disciplines and provides a coherent model of the world to the reader or viewer. Recent advancement in machine learning and Large Language Models (LLMs) have enable great strides in analyzing natural language. However, Large language models (LLMs) still struggle with complex narrative arcs as well as narratives containing conflicting information. Recent work indicates LLMs augmented with external knowledge bases can improve the accuracy and interpretability of the resulting models. In this work, we analyze the effectiveness of applying knowledge graphs (KGs) in understanding true-crime podcast data from both classical Natural Language Processing (NLP) and LLM approaches. We directly compare KG-augmented LLMs (KGLLMs) with classical methods for KG construction, topic modeling, and sentiment analysis. Additionally, the KGLLM allows us to query the knowledge base in natural language and test its ability to factually answer questions. We examine the robustness of the model to adversarial prompting in order to test the model's ability to deal with conflicting information. Finally, we apply classical methods to understand more subtle aspects of the text such as the use of hearsay and sentiment in narrative construction and propose future directions. Our results indicate that KGLLMs outperform LLMs on a variety of metrics, are more robust to adversarial prompts, and are more capable of summarizing the text into topics.
• 摘要：叙事数据涵盖所有学科，并为读者或观众提供连贯的世界模型。机器学习和大型语言模型 (LLM) 的最新进展使自然语言分析取得了长足进步。然而，大型语言模型 (LLM) 仍然难以处理复杂的叙事弧线以及包含冲突信息的叙事。最近的研究表明，使用外部知识库增强的 LLM 可以提高结果模型的准确性和可解释性。在这项工作中，我们分析了从经典自然语言处理 (NLP) 和 LLM 方法中应用知识图谱 (KG) 理解真实犯罪播客数据的有效性。我们直接将 KG 增强 LLM (KGLLM) 与经典的 KG 构建、主题建模和情感分析方法进行比较。此外，KGLLM 允许我们用自然语言查询知识库并测试其回答问题的能力。我们检查模型对对抗性提示的鲁棒性，以测试模型处理冲突信息的能力。最后，我们采用经典方法来理解文本中更微妙的方面，例如在叙事结构中使用传闻和情感，并提出未来的发展方向。我们的结果表明，KGLLM 在各种指标上都优于 LLM，对对抗性提示更具鲁棒性，并且更有能力将文本概括为主题。

Title: TODO: Enhancing LLM Alignment with Ternary Preferences

• Authors: Yuxiang Guo, Lu Yin, Bo Jiang, Jiaqi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02442
• Pdf URL: https://arxiv.org/pdf/2411.02442
• Copy Paste: [[2411.02442]] TODO: Enhancing LLM Alignment with Ternary Preferences(https://arxiv.org/abs/2411.02442)
• Keywords: language model, llm
• Abstract: Aligning large language models (LLMs) with human intent is critical for enhancing their performance across a variety of tasks. Standard alignment techniques, such as Direct Preference Optimization (DPO), often rely on the binary Bradley-Terry (BT) model, which can struggle to capture the complexities of human preferences -- particularly in the presence of noisy or inconsistent labels and frequent ties. To address these limitations, we introduce the Tie-rank Oriented Bradley-Terry model (TOBT), an extension of the BT model that explicitly incorporates ties, enabling more nuanced preference representation. Building on this, we propose Tie-rank Oriented Direct Preference Optimization (TODO), a novel alignment algorithm that leverages TOBT's ternary ranking system to improve preference alignment. In evaluations on Mistral-7B and Llama 3-8B models, TODO consistently outperforms DPO in modeling preferences across both in-distribution and out-of-distribution datasets. Additional assessments using MT Bench and benchmarks such as Piqa, ARC-c, and MMLU further demonstrate TODO's superior alignment performance. Notably, TODO also shows strong results in binary preference alignment, highlighting its versatility and potential for broader integration into LLM alignment. The implementation details can be found in this https URL.
• 摘要：将大型语言模型 (LLM) 与人类意图对齐对于提高其在各种任务中的表现至关重要。标准对齐技术，例如直接偏好优化 (DPO)，通常依赖于二元 Bradley-Terry (BT) 模型，该模型很难捕捉人类偏好的复杂性——尤其是在存在嘈杂或不一致的标签和频繁联系的情况下。为了解决这些限制，我们引入了 Tie-rank 导向 Bradley-Terry 模型 (TOBT)，这是 BT 模型的扩展，明确纳入了联系，从而能够实现更细致入微的偏好表示。在此基础上，我们提出了 Tie-rank 导向直接偏好优化 (TODO)，这是一种新颖的对齐算法，它利用 TOBT 的三元排名系统来改进偏好对齐。在对 Mistral-7B 和 Llama 3-8B 模型的评估中，TODO 在分布内和分布外数据集中建模偏好方面始终优于 DPO。使用 MT Bench 和 Piqa、ARC-c 和 MMLU 等基准进行的额外评估进一步证明了 TODO 的卓越对齐性能。值得注意的是，TODO 在二进制偏好对齐中也表现出色，凸显了其多功能性和更广泛地集成到 LLM 对齐中的潜力。实现细节可在此 https URL 中找到。

Title: Rate, Explain and Cite (REC): Enhanced Explanation and Attribution in Automatic Evaluation by Large Language Models

• Authors: Aliyah R. Hsu, James Zhu, Zhichao Wang, Bin Bi, Shubham Mehrotra, Shiva K. Pentyala, Katherine Tan, Xiang-Bo Mao, Roshanak Omrani, Sougata Chaudhuri, Regunathan Radhakrishnan, Sitaram Asur, Claire Na Cheng, Bin Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02448
• Pdf URL: https://arxiv.org/pdf/2411.02448
• Copy Paste: [[2411.02448]] Rate, Explain and Cite (REC): Enhanced Explanation and Attribution in Automatic Evaluation by Large Language Models(https://arxiv.org/abs/2411.02448)
• Keywords: language model, llm, hallucination
• Abstract: LLMs have demonstrated impressive proficiency in generating coherent and high-quality text, making them valuable across a range of text-generation tasks. However, rigorous evaluation of this generated content is crucial, as ensuring its quality remains a significant challenge due to persistent issues such as factual inaccuracies and hallucinations. This paper introduces two fine-tuned general-purpose LLM autoevaluators, REC-12B and REC-70B, specifically designed to evaluate generated text across several dimensions: faithfulness, instruction following, coherence, and completeness. These models not only provide ratings for these metrics but also offer detailed explanations and verifiable citations, thereby enhancing trust in the content. Moreover, the models support various citation modes, accommodating different requirements for latency and granularity. Extensive evaluations on diverse benchmarks demonstrate that our general-purpose LLM auto-evaluator, REC-70B, outperforms state-of-the-art LLMs, excelling in content evaluation by delivering better quality explanations and citations with minimal bias. It achieves Rank \#1 as a generative model on the RewardBench leaderboard\footnote{\url{this https URL}} under the model name \texttt{TextEval-Llama3.1-70B}. Our REC dataset and models are released at \url{this https URL}.
• 摘要：LLM 在生成连贯且高质量的文本方面表现出色，在一系列文本生成任务中都具有重要价值。然而，对生成的内容进行严格评估至关重要，因为由于事实不准确和幻觉等持续存在的问题，确保其质量仍然是一项重大挑战。本文介绍了两个经过微调的通用 LLM 自动评估器 REC-12B 和 REC-70B，专门用于从多个维度评估生成的文本：忠实度、指令遵循度、连贯性和完整性。这些模型不仅为这些指标提供评级，还提供详细的解释和可验证的引用，从而增强对内容的信任。此外，这些模型支持各种引用模式，可满足不同的延迟和粒度要求。对各种基准的广泛评估表明，我们的通用 LLM 自动评估器 REC-70B 的表现优于最先进的 LLM，通过提供更高质量的解释和引用并以最小的偏见在内容评估方面表现出色。它在 RewardBench 排行榜\footnote{\url{此 https URL}} 上以模型名称 \texttt{TextEval-Llama3.1-70B} 获得了排名第一的生成模型。我们的 REC 数据集和模型在 \url{此 https URL} 上发布。

Title: High-performance automated abstract screening with large language model ensembles

• Authors: Rohan Sanghera, Arun James Thirunavukarasu, Marc El Khoury, Jessica O'Logbon, Yuqing Chen, Archie Watt, Mustafa Mahmood, Hamid Butt, George Nishimura, Andrew Soltan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.02451
• Pdf URL: https://arxiv.org/pdf/2411.02451
• Copy Paste: [[2411.02451]] High-performance automated abstract screening with large language model ensembles(https://arxiv.org/abs/2411.02451)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large language models (LLMs) excel in tasks requiring processing and interpretation of input text. Abstract screening is a labour-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies identified by a literature search. Here, LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialled on systematic reviews in a full issue of the Cochrane Library to evaluate their accuracy in zero-shot binary classification for abstract screening. Trials over a subset of 800 records identified optimal prompting strategies and demonstrated superior performance of LLMs to human researchers in terms of sensitivity (LLMmax = 1.000, humanmax = 0.775), precision (LLMmax = 0.927, humanmax = 0.911), and balanced accuracy (LLMmax = 0.904, humanmax = 0.865). The best performing LLM-prompt combinations were trialled across every replicated search result (n = 119,691), and exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096). 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458, with less observed performance drop in larger trials. Significant variation in performance was observed between reviews, highlighting the importance of domain-specific validation before deployment. LLMs may reduce the human labour cost of systematic review with maintained or improved accuracy and sensitivity. Systematic review is the foundation of evidence-based medicine, and LLMs can contribute to increasing the efficiency and quality of this mode of research.
• 摘要：大型语言模型 (LLM) 在需要处理和解释输入文本的任务中表现出色。摘要筛选是系统评价中一项劳动密集型的工作，涉及对文献检索确定的大量研究重复应用纳入和排除标准。在这里，我们在 Cochrane Library 的一期完整版系统评价中试用了 LLM（GPT-3.5 Turbo、GPT-4 Turbo、GPT-4o、Llama 3 70B、Gemini 1.5 Pro 和 Claude Sonnet 3.5），以评估它们在摘要筛选的零样本二元分类中的准确性。对 800 条记录子集的试验确定了最佳提示策略，并证明了 LLM 在灵敏度（LLMmax = 1.000，humanmax = 0.775）、精确度（LLMmax = 0.927，humanmax = 0.911）和平衡准确度（LLMmax = 0.904，humanmax = 0.865）方面优于人类研究人员。在每次重复的搜索结果（n = 119,691）中试用了表现最佳的 LLM-prompt 组合，结果显示灵敏度一致（范围 0.756-1.000），但精确度降低（范围 0.004-0.096）。66 个 LLM-human 和 LLM-LLM 集成表现出完美的灵敏度，最大精确度为 0.458，在更大规模的试验中性能下降较少。在评价之间观察到了显著的性能差异，凸显了部署前进行领域特定验证的重要性。LLM 可以降低系统评价的人力成本，同时保持或提高精确度和灵敏度。系统评价是循证医学的基础，LLM 有助于提高这种研究模式的效率和质量。

Title: Graph-based Confidence Calibration for Large Language Models

• Authors: Yukun Li, Sijia Wang, Lifu Huang, Li-Ping Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02454
• Pdf URL: https://arxiv.org/pdf/2411.02454
• Copy Paste: [[2411.02454]] Graph-based Confidence Calibration for Large Language Models(https://arxiv.org/abs/2411.02454)
• Keywords: language model, llm
• Abstract: One important approach to improving the reliability of large language models (LLMs) is to provide accurate confidence estimations regarding the correctness of their answers. However, developing a well-calibrated confidence estimation model is challenging, as mistakes made by LLMs can be difficult to detect. We propose a novel method combining the LLM's self-consistency with labeled data and training an auxiliary model to estimate the correctness of its responses to questions. This auxiliary model predicts the correctness of responses based solely on their consistent information. To set up the learning problem, we use a weighted graph to represent the consistency among the LLM's multiple responses to a question. Correctness labels are assigned to these responses based on their similarity to the correct answer. We then train a graph neural network to estimate the probability of correct responses. Experiments demonstrate that the proposed approach substantially outperforms several of the most recent methods in confidence calibration across multiple widely adopted benchmark datasets. Furthermore, the proposed approach significantly improves the generalization capability of confidence calibration on out-of-domain (OOD) data.
• 摘要：提高大型语言模型 (LLM) 可靠性的一个重要方法是提供关于其答案正确性的准确置信度估计。然而，开发一个经过良好校准的置信度估计模型具有挑战性，因为 LLM 所犯的错误很难被发现。我们提出了一种新方法，将 LLM 的自洽性与标记数据相结合，并训练辅助模型来估计其对问题的回答的正确性。该辅助模型仅根据其一致性信息来预测回答的正确性。为了设置学习问题，我们使用加权图来表示 LLM 对问题的多个回答之间的一致性。根据这些回答与正确答案的相似性为它们分配正确性标签。然后，我们训练一个图神经网络来估计正确回答的概率。实验表明，在多个广泛采用的基准数据集的置信度校准中，所提出的方法大大优于几种最新的方法。此外，所提出的方法显著提高了域外 (OOD) 数据的置信度校准的泛化能力。

Title: An Exploration of Higher Education Course Evaluation by Large Language Models

• Authors: Bo Yuan, Jiazi Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02455
• Pdf URL: https://arxiv.org/pdf/2411.02455
• Copy Paste: [[2411.02455]] An Exploration of Higher Education Course Evaluation by Large Language Models(https://arxiv.org/abs/2411.02455)
• Keywords: language model, llm, prompt
• Abstract: Course evaluation is a critical component in higher education pedagogy. It not only serves to identify limitations in existing course designs and provide a basis for curricular innovation, but also to offer quantitative insights for university administrative decision-making. Traditional evaluation methods, primarily comprising student surveys, instructor self-assessments, and expert reviews, often encounter challenges, including inherent subjectivity, feedback delays, inefficiencies, and limitations in addressing innovative teaching approaches. Recent advancements in large language models (LLMs) within artificial intelligence (AI) present promising new avenues for enhancing course evaluation processes. This study explores the application of LLMs in automated course evaluation from multiple perspectives and conducts rigorous experiments across 100 courses at a major university in China. The findings indicate that: (1) LLMs can be an effective tool for course evaluation; (2) their effectiveness is contingent upon appropriate fine-tuning and prompt engineering; and (3) LLM-generated evaluation results demonstrate a notable level of rationality and interpretability.
• 摘要：课程评估是高等教育教学中的一个重要组成部分。它不仅可以发现现有课程设计的局限性并为课程创新提供依据，还可以为大学行政决策提供定量见解。传统的评估方法主要包括学生调查、教师自我评估和专家评审，这些方法经常遇到挑战，包括固有的主观性、反馈延迟、效率低下以及在解决创新教学方法方面的局限性。人工智能 (AI) 领域大型语言模型 (LLM) 的最新进展为增强课程评估流程提供了有希望的新途径。本研究从多个角度探讨了 LLM 在自动化课程评估中的应用，并在中国一所主要大学的 100 门课程中进行了严格的实验。研究结果表明：(1) LLM 可以成为课程评估的有效工具；(2) 其有效性取决于适当的微调和及时的工程设计；(3) LLM 生成的评估结果具有显著的合理性和可解释性。

Title: A Multi-Task Role-Playing Agent Capable of Imitating Character Linguistic Styles

• Authors: Siyuan Chen, Qingyi Si, Chenxu Yang, Yunzhi Liang, Zheng Lin, Huan Liu, Weiping Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02457
• Pdf URL: https://arxiv.org/pdf/2411.02457
• Copy Paste: [[2411.02457]] A Multi-Task Role-Playing Agent Capable of Imitating Character Linguistic Styles(https://arxiv.org/abs/2411.02457)
• Keywords: language model, llm, agent
• Abstract: The advent of large language models (LLMs) has significantly propelled the advancement of Role-Playing Agents (RPAs). However, current Role-Playing Agents predominantly focus on mimicking a character's fundamental attributes while neglecting the replication of linguistic style, and they are incapable of effectively replicating characters when performing tasks beyond multi-turn dialogues, which results in generated responses that lack authenticity. The reason current RPAs lack this capability is due to the nature of existing character datasets, which lack collections of character quotations and are limited to multi-turn dialogue tasks, constraining the RPA's performance across other task domains and failing to mimic a character's linguistic style. To address this gap, we developed a multi-task role-playing dataset named MRstyle, which encompasses a substantial number of real individuals along with their quotations and covers seven different tasks. On this basis, we develop StyleRPA, a Multi-Task Role-Playing Agent (MRPA) that significantly outperforms recent open-source LLMs and RPAs baselines on 7 tasks including Dialogue, Dictionary, Composition, Story Generation, Product Description, Music Commentary, and Open Question Answering. The code and data will be released.
• 摘要：大型语言模型 (LLM) 的出现极大地推动了角色扮演代理 (RPA) 的发展。然而，目前的角色扮演代理主要关注模仿角色的基本属性，而忽略了语言风格的复制，并且在执行多轮对话以外的任务时无法有效地复制角色，从而导致生成的响应缺乏真实性。当前 RPA 缺乏这种能力的原因在于现有角色数据集的性质，这些数据集缺乏角色引语集合并且仅限于多轮对话任务，这限制了 RPA 在其他任务领域的性能，并且无法模仿角色的语言风格。为了弥补这一差距，我们开发了一个名为 MRstyle 的多任务角色扮演数据集，它包含大量真实个体及其引语，并涵盖七种不同的任务。在此基础上，我们开发了 StyleRPA，一个多任务角色扮演代理 (MRPA)，它在对话、词典、作文、故事生成、产品描述、音乐评论和开放式问答等 7 个任务上的表现显著优于最近的开源 LLM 和 RPA 基线。代码和数据即将发布。

Title: Code-Switching Curriculum Learning for Multilingual Transfer in LLMs

• Authors: Haneul Yoo, Cheonbok Park, Sangdoo Yun, Alice Oh, Hwaran Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02460
• Pdf URL: https://arxiv.org/pdf/2411.02460
• Copy Paste: [[2411.02460]] Code-Switching Curriculum Learning for Multilingual Transfer in LLMs(https://arxiv.org/abs/2411.02460)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) now exhibit near human-level performance in various tasks, but their performance drops drastically after a handful of high-resource languages due to the imbalance in pre-training data. Inspired by the human process of second language acquisition, particularly code-switching (the practice of language alternation in a conversation), we propose code-switching curriculum learning (CSCL) to enhance cross-lingual transfer for LLMs. CSCL mimics the stages of human language learning by progressively training models with a curriculum consisting of 1) token-level code-switching, 2) sentence-level code-switching, and 3) monolingual corpora. Using Qwen 2 as our underlying model, we demonstrate the efficacy of the CSCL in improving language transfer to Korean, achieving significant performance gains compared to monolingual continual pre-training methods. Ablation studies reveal that both token- and sentence-level code-switching significantly enhance cross-lingual transfer and that curriculum learning amplifies these effects. We also extend our findings into various languages, including Japanese (high-resource) and Indonesian (low-resource), and using two additional models (Gemma 2 and Phi 3.5). We further show that CSCL mitigates spurious correlations between language resources and safety alignment, presenting a robust, efficient framework for more equitable language transfer in LLMs. We observe that CSCL is effective for low-resource settings where high-quality, monolingual corpora for language transfer are hardly available.
• 摘要：大型语言模型 (LLM) 现在在各种任务中都表现出接近人类水平的性能，但由于预训练数据的不平衡，它们的性能在少数高资源语言之后急剧下降。受人类第二语言习得过程的启发，特别是代码转换（对话中的语言交替实践），我们提出了代码转换课程学习 (CSCL) 来增强 LLM 的跨语言迁移。CSCL 通过使用由 1) 标记级代码转换、2) 句子级代码转换和 3) 单语语料库组成的课程逐步训练模型来模仿人类语言学习的各个阶段。使用 Qwen 2 作为我们的基础模型，我们证明了 CSCL 在改善向韩语的语言迁移方面的有效性，与单语持续预训练方法相比，取得了显着的性能提升。消融研究表明，标记级和句子级代码转换都显着增强了跨语言迁移，并且课程学习会放大这些效果。我们还将研究结果扩展到各种语言，包括日语（资源丰富）和印尼语（资源匮乏），并使用了另外两个模型（Gemma 2 和 Phi 3.5）。我们进一步表明，CSCL 减轻了语言资源与安全一致性之间的虚假相关性，为 LLM 中更公平的语言迁移提供了一个强大而有效的框架。我们观察到，CSCL 适用于资源匮乏的环境，因为在这种环境中，高质量的单语语料库几乎不可用。

Title: Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control

• Authors: Yuxin Xiao, Chaoqun Wan, Yonggang Zhang, Wenxiao Wang, Binbin Lin, Xiaofei He, Xu Shen, Jieping Ye
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02461
• Pdf URL: https://arxiv.org/pdf/2411.02461
• Copy Paste: [[2411.02461]] Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control(https://arxiv.org/abs/2411.02461)
• Keywords: language model, llm
• Abstract: As the development and application of Large Language Models (LLMs) continue to advance rapidly, enhancing their trustworthiness and aligning them with human preferences has become a critical area of research. Traditional methods rely heavily on extensive data for Reinforcement Learning from Human Feedback (RLHF), but representation engineering offers a new, training-free approach. This technique leverages semantic features to control the representation of LLM's intermediate hidden states, enabling the model to meet specific requirements such as increased honesty or heightened safety awareness. However, a significant challenge arises when attempting to fulfill multiple requirements simultaneously. It proves difficult to encode various semantic contents, like honesty and safety, into a singular semantic feature, restricting its practicality. In this work, we address this issue through ``Sparse Activation Control''. By delving into the intrinsic mechanisms of LLMs, we manage to identify and pinpoint components that are closely related to specific tasks within the model, i.e., attention heads. These heads display sparse characteristics that allow for near-independent control over different tasks. Our experiments, conducted on the open-source Llama series models, have yielded encouraging results. The models were able to align with human preferences on issues of safety, factuality, and bias concurrently.
• 摘要：随着大型语言模型 (LLM) 的开发和应用继续快速发展，提高其可信度并使其与人类偏好保持一致已成为一个关键的研究领域。传统方法严重依赖大量数据来进行人类反馈强化学习 (RLHF)，但表示工程提供了一种无需训练的新方法。该技术利用语义特征来控制 LLM 中间隐藏状态的表示，使模型能够满足特定要求，例如提高诚实度或增强安全意识。然而，当试图同时满足多个要求时，就会出现一个重大挑战。事实证明，将各种语义内容（如诚实和安全）编码为单一的语义特征非常困难，这限制了它的实用性。在这项工作中，我们通过“稀疏激活控制”解决了这个问题。通过深入研究 LLM 的内在机制，我们设法识别并确定与模型中特定任务密切相关的组件，即注意力头。这些头表现出稀疏特性，允许对不同任务进行近乎独立的控制。我们在开源 Llama 系列模型上进行的实验取得了令人鼓舞的结果。这些模型能够同时在安全性、事实性和偏见问题上与人类的偏好保持一致。

Title: A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification

• Authors: Sorouralsadat Fatemi, Yuheng Hu, Maryam Mousavi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02476
• Pdf URL: https://arxiv.org/pdf/2411.02476
• Copy Paste: [[2411.02476]] A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification(https://arxiv.org/abs/2411.02476)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across diverse Natural Language Processing (NLP) tasks, including language understanding, reasoning, and generation. However, general-domain LLMs often struggle with financial tasks due to the technical and specialized nature of financial texts. This study investigates the efficacy of instruction fine-tuning smaller-scale LLMs, including Mistral-7B, Llama3-8B, and Phi3-mini, to enhance their performance in financial text classification tasks. We fine-tuned both instruction-tuned and base models across four financial classification tasks, achieving significant improvements in task-specific performance. Furthermore, we evaluated the zero-shot capabilities of these fine-tuned models on three unseen complex financial tasks, including argument classification, deal completeness classification, and causal classification. Our results indicate while base model fine-tuning led to greater degradation, instruction-tuned models maintained more robust performance. To address this degradation, we employed model merging techniques, integrating single-task domain-specific fine-tuned models with the base model. Using this merging method resulted in significant enhancements in zero-shot performance, even exceeding the original model's accuracy on certain datasets. Our findings underscore the effectiveness of instruction fine-tuning and model merging for adapting LLMs to specialized financial text classification tasks.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理 (NLP) 任务中表现出令人印象深刻的能力，包括语言理解、推理和生成。然而，由于金融文本的技术性和专业性，通用领域的 LLM 通常在金融任务中遇到困难。本研究调查了指令微调小型 LLM（包括 Mistral-7B、Llama3-8B 和 Phi3-mini）在提高金融文本分类任务性能方面的有效性。我们在四个金融分类任务中对指令调整模型和基础模型进行了微调，在任务特定性能方面取得了显著的改进。此外，我们在三个看不见的复杂金融任务上评估了这些微调模型的零样本能力，包括参数分类、交易完整性分类和因果分类。我们的结果表明，虽然基础模型微调导致性能下降幅度更大，但指令调整模型保持了更稳健的性能。为了解决这种性能下降问题，我们采用了模型合并技术，将单任务领域特定微调模型与基础模型集成在一起。使用这种合并方法可以显著提高零样本性能，甚至在某些数据集上超过了原始模型的准确率。我们的研究结果强调了指令微调和模型合并对于使 LLM 适应专门的金融文本分类任务的有效性。

Title: Fantastic LLMs for Preference Data Annotation and How to (not) Find Them

• Authors: Guangxuan Xu, Kai Xu, Shivchander Sudalairaj, Hao Wang, Akash Srivastava
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02481
• Pdf URL: https://arxiv.org/pdf/2411.02481
• Copy Paste: [[2411.02481]] Fantastic LLMs for Preference Data Annotation and How to (not) Find Them(https://arxiv.org/abs/2411.02481)
• Keywords: language model, llm
• Abstract: Preference tuning of large language models (LLMs) relies on high-quality human preference data, which is often expensive and time-consuming to gather. While existing methods can use trained reward models or proprietary model as judges for preference annotation, they have notable drawbacks: training reward models remain dependent on initial human data, and using proprietary model imposes license restrictions that inhibits commercial usage. In this paper, we introduce customized density ratio (CDR) that leverages open-source LLMs for data annotation, offering an accessible and effective solution. Our approach uses the log-density ratio between a well-aligned LLM and a less aligned LLM as a reward signal. We explores 221 different LLMs pairs and empirically demonstrate that increasing the performance gap between paired LLMs correlates with better reward generalization. Furthermore, we show that tailoring the density ratio reward function with specific criteria and preference exemplars enhances performance across domains and within target areas. In our experiment using density ratio from a pair of Mistral-7B models, CDR achieves a RewardBench score of 82.6, outperforming the best in-class trained reward functions and demonstrating competitive performance against SoTA models in Safety (91.0) and Reasoning (88.0) domains. We use CDR to annotate an on-policy preference dataset with which we preference tune Llama-3-8B-Instruct with SimPO. The final model achieves a 37.4% (+15.1%) win rate on ArenaHard and a 40.7% (+17.8%) win rate on Length-Controlled AlpacaEval 2.0, along with a score of 8.0 on MT-Bench.
• 摘要：大型语言模型 (LLM) 的偏好调整依赖于高质量的人类偏好数据，而这些数据的收集通常既昂贵又耗时。虽然现有方法可以使用训练有素的奖励模型或专有模型作为偏好注释的判断标准，但它们存在明显的缺点：训练奖励模型仍然依赖于初始人类数据，而使用专有模型会施加许可限制，从而抑制商业使用。在本文中，我们引入了定制密度比 (CDR)，利用开源 LLM 进行数据注释，提供了一种可访问且有效的解决方案。我们的方法使用对齐良好的 LLM 和对齐程度较低的 LLM 之间的对数密度比作为奖励信号。我们探索了 221 种不同的 LLM 对，并通过经验证明，成对的 LLM 之间的性能差距越大，奖励泛化效果越好。此外，我们表明，使用特定标准和偏好样本定制密度比奖励函数可以提高跨领域和目标区域内的性能。在我们使用一对 Mistral-7B 模型的密度比进行的实验中，CDR 的 RewardBench 得分为 82.6，优于同类最佳的训练奖励函数，并且在安全 (91.0) 和推理 (88.0) 领域表现出与 SoTA 模型相当的竞争力。我们使用 CDR 注释一个基于策略的偏好数据集，我们利用该数据集使用 SimPO 对 Llama-3-8B-Instruct 进行偏好调整。最终模型在 ArenaHard 上的胜率为 37.4% (+15.1%)，在 Length-Controlled AlpacaEval 2.0 上的胜率为 40.7% (+17.8%)，在 MT-Bench 上的得分为 8.0。

Title: Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes

• Authors: Balu Bhasuran, Qiao Jin, Yuzhang Xie, Carl Yang, Karim Hanna, Jennifer Costa, Cindy Shavor, Zhiyong Lu, Zhe He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02523
• Pdf URL: https://arxiv.org/pdf/2411.02523
• Copy Paste: [[2411.02523]] Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes(https://arxiv.org/abs/2411.02523)
• Keywords: language model, gpt, llm
• Abstract: Differential diagnosis is crucial for medicine as it helps healthcare providers systematically distinguish between conditions that share similar symptoms. This study assesses the impact of lab test results on differential diagnoses (DDx) made by large language models (LLMs). Clinical vignettes from 50 case reports from PubMed Central were created incorporating patient demographics, symptoms, and lab results. Five LLMs GPT-4, GPT-3.5, Llama-2-70b, Claude-2, and Mixtral-8x7B were tested to generate Top 10, Top 5, and Top 1 DDx with and without lab data. A comprehensive evaluation involving GPT-4, a knowledge graph, and clinicians was conducted. GPT-4 performed best, achieving 55% accuracy for Top 1 diagnoses and 60% for Top 10 with lab data, with lenient accuracy up to 80%. Lab results significantly improved accuracy, with GPT-4 and Mixtral excelling, though exact match rates were low. Lab tests, including liver function, metabolic/toxicology panels, and serology/immune tests, were generally interpreted correctly by LLMs for differential diagnosis.
• 摘要：鉴别诊断对医学至关重要，因为它可以帮助医疗保健提供者系统地区分具有相似症状的疾病。这项研究评估了实验室测试结果对大型语言模型 (LLM) 做出的鉴别诊断 (DDx) 的影响。从 PubMed Central 的 50 份病例报告中创建了临床案例，结合了患者的人口统计、症状和实验室结果。测试了五个 LLM GPT-4、GPT-3.5、Llama-2-70b、Claude-2 和 Mixtral-8x7B，以生成带有和不带有实验室数据的 Top 10、Top 5 和 Top 1 DDx。进行了涉及 GPT-4、知识图谱和临床医生的全面评估。GPT-4 表现最佳，使用实验室数据时 Top 1 诊断的准确率为 55%，Top 10 诊断的准确率为 60%，宽松准确率高达 80%。实验室结果显著提高了准确率，GPT-4 和 Mixtral 表现出色，但完全匹配率很低。实验室测试，包括肝功能、代谢/毒理学小组以及血清学/免疫测试，通常都能被法学硕士正确解释以进行鉴别诊断。

Title: What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length

• Authors: Lindia Tjuatja, Graham Neubig, Tal Linzen, Sophie Hao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.02528
• Pdf URL: https://arxiv.org/pdf/2411.02528
• Copy Paste: [[2411.02528]] What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length(https://arxiv.org/abs/2411.02528)
• Keywords: language model
• Abstract: When comparing the linguistic capabilities of language models (LMs) with humans using LM probabilities, factors such as the length of the sequence and the unigram frequency of lexical items have a significant effect on LM probabilities in ways that humans are largely robust to. Prior works in comparing LM and human acceptability judgments treat these effects uniformly across models, making a strong assumption that models require the same degree of adjustment to control for length and unigram frequency effects. We propose MORCELA, a new linking theory between LM scores and acceptability judgments where the optimal level of adjustment for these effects is estimated from data via learned parameters for length and unigram frequency. We first show that MORCELA outperforms a commonly used linking theory for acceptability--SLOR (Pauls and Klein, 2012; Lau et al. 2017)--across two families of transformer LMs (Pythia and OPT). Furthermore, we demonstrate that the assumed degrees of adjustment in SLOR for length and unigram frequency overcorrect for these confounds, and that larger models require a lower relative degree of adjustment for unigram frequency, though a significant amount of adjustment is still necessary for all models. Finally, our subsequent analysis shows that larger LMs' lower susceptibility to frequency effects can be explained by an ability to better predict rarer words in context.
• 摘要：当使用 LM 概率将语言模型 (LM) 的语言能力与人类进行比较时，诸如序列长度和词汇项的单字频率等因素对 LM 概率有显著影响，而人类对此的影响则基本不受影响。在比较 LM 和人类可接受性判断的先前研究中，这些影响在各个模型中是统一对待的，并强烈假设模型需要相同程度的调整来控制长度和单字频率的影响。我们提出了 MORCELA，这是一种 LM 分数和可接受性判断之间的新链接理论，其中这些影响的最佳调整水平是通过学习到的长度和单字频率参数从数据中估计出来的。我们首先表明，在两大系列的 Transformer LM（Pythia 和 OPT）中，MORCELA 的表现优于一种常用的可接受性链接理论 SLOR（Pauls 和 Klein，2012 年；Lau 等人，2017 年）。此外，我们证明了 SLOR 中针对长度和单字频率的假定调整程度对这些混淆因素进行了过度校正，并且较大的模型需要相对较低的单字频率调整程度，尽管所有模型仍然需要进行大量调整。最后，我们随后的分析表明，较大的 LM 对频率效应的敏感性较低，这可以通过更好地预测上下文中罕见单词的能力来解释。

Title: Towards Leveraging News Media to Support Impact Assessment of AI Technologies

• Authors: Mowafak Allaham, Kimon Kieslich, Nicholas Diakopoulos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02536
• Pdf URL: https://arxiv.org/pdf/2411.02536
• Copy Paste: [[2411.02536]] Towards Leveraging News Media to Support Impact Assessment of AI Technologies(https://arxiv.org/abs/2411.02536)
• Keywords: gpt, llm
• Abstract: Expert-driven frameworks for impact assessments (IAs) may inadvertently overlook the effects of AI technologies on the public's social behavior, policy, and the cultural and geographical contexts shaping the perception of AI and the impacts around its use. This research explores the potentials of fine-tuning LLMs on negative impacts of AI reported in a diverse sample of articles from 266 news domains spanning 30 countries around the world to incorporate more diversity into IAs. Our findings highlight (1) the potential of fine-tuned open-source LLMs in supporting IA of AI technologies by generating high-quality negative impacts across four qualitative dimensions: coherence, structure, relevance, and plausibility, and (2) the efficacy of small open-source LLM (Mistral-7B) fine-tuned on impacts from news media in capturing a wider range of categories of impacts that GPT-4 had gaps in covering.
• 摘要：专家驱动的影响评估 (IA) 框架可能会无意中忽视人工智能技术对公众社会行为、政策以及塑造人工智能认知和其使用影响的文化和地理背景的影响。这项研究探讨了对来自全球 30 个国家/地区的 266 个新闻领域的多样化文章样本中报道的人工智能负面影响进行微调的 LLM 的潜力，以将更多多样性纳入 IA。我们的研究结果强调了 (1) 微调的开源 LLM 在支持人工智能技术 IA 方面的潜力，通过在四个定性维度上产生高质量的负面影响：连贯性、结构性、相关性和合理性，以及 (2) 针对新闻媒体影响进行微调的小型开源 LLM (Mistral-7B) 在捕捉 GPT-4 无法覆盖的更广泛影响类别方面的有效性。

Title: MILU: A Multi-task Indic Language Understanding Benchmark

• Authors: Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, Jaydeep Sen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.02538
• Pdf URL: https://arxiv.org/pdf/2411.02538
• Copy Paste: [[2411.02538]] MILU: A Multi-task Indic Language Understanding Benchmark(https://arxiv.org/abs/2411.02538)
• Keywords: language model, gpt, llm
• Abstract: Evaluating Large Language Models (LLMs) in low-resource and linguistically diverse languages remains a significant challenge in NLP, particularly for languages using non-Latin scripts like those spoken in India. Existing benchmarks predominantly focus on English, leaving substantial gaps in assessing LLM capabilities in these languages. We introduce MILU, a Multi task Indic Language Understanding Benchmark, a comprehensive evaluation benchmark designed to address this gap. MILU spans 8 domains and 42 subjects across 11 Indic languages, reflecting both general and culturally specific knowledge. With an India-centric design, incorporates material from regional and state-level examinations, covering topics such as local history, arts, festivals, and laws, alongside standard subjects like science and mathematics. We evaluate over 42 LLMs, and find that current LLMs struggle with MILU, with GPT-4o achieving the highest average accuracy at 72 percent. Open multilingual models outperform language-specific fine-tuned models, which perform only slightly better than random baselines. Models also perform better in high resource languages as compared to low resource ones. Domain-wise analysis indicates that models perform poorly in culturally relevant areas like Arts and Humanities, Law and Governance compared to general fields like STEM. To the best of our knowledge, MILU is the first of its kind benchmark focused on Indic languages, serving as a crucial step towards comprehensive cultural evaluation. All code, benchmarks, and artifacts will be made publicly available to foster open research.
• 摘要：在资源匮乏且语言多样化的语言中评估大型语言模型 (LLM) 仍然是 NLP 领域的一项重大挑战，尤其是对于使用非拉丁字母的语言，例如印度使用的语言。现有的基准主要集中在英语上，在评估这些语言的 LLM 能力方面存在很大差距。我们推出了 MILU，这是一种多任务印度语言理解基准，旨在解决这一差距的综合评估基准。MILU 涵盖 11 种印度语言的 8 个领域和 42 个科目，反映了一般和文化特定的知识。采用以印度为中心的设计，结合了地区和州级考试的材料，涵盖当地历史、艺术、节日和法律等主题，以及科学和数学等标准科目。我们评估了 42 个以上的 LLM，发现当前的 LLM 在 MILU 方面表现不佳，GPT-4o 的平均准确率最高，为 72%。开放多语言模型优于特定语言的微调模型，后者的表现仅略好于随机基线。与资源较少的语言相比，模型在资源丰富的语言中的表现也更好。领域分析表明，与 STEM 等一般领域相比，模型在艺术和人文、法律和治理等文化相关领域表现不佳。据我们所知，MILU 是第一个专注于印度语的同类基准，是实现全面文化评估的关键一步。所有代码、基准和工件都将公开，以促进开放研究。

Title: MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

• Authors: Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, Wei Ping
• Subjects: cs.CL, cs.AI, cs.CV, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02571
• Pdf URL: https://arxiv.org/pdf/2411.02571
• Copy Paste: [[2411.02571]] MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs(https://arxiv.org/abs/2411.02571)
• Keywords: language model, llm, prompt
• Abstract: State-of-the-art retrieval models typically address a straightforward search scenario, where retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but underperforms a smaller CLIP retriever in cross-modal retrieval tasks due to modality bias from MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose to continually fine-tune the universal multimodal retriever to enhance its text retrieval capability while maintaining multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on MTEB retrieval benchmark. Finally, we explore to prompt the off-the-shelf MLLMs as the zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way to advance universal multimodal retrieval in the future.
• 摘要：最先进的检索模型通常解决的是一个简单的搜索场景，其中检索任务是固定的（例如，找到一段话来回答一个特定的问题），并且查询和检索结果都只支持单一模态。本文介绍了使用多模态大型语言模型 (MLLM) 推进信息检索的技术，从而实现更广泛的搜索场景，称为通用多模态检索，其中可容纳多种模态和不同的检索任务。为此，我们首先研究在 10 个数据集上使用 16 个检索任务对 MLLM 作为双编码器检索器进行微调。我们的实证结果表明，经过微调的 MLLM 检索器能够理解由文本和图像组成的具有挑战性的查询，但由于 MLLM 的模态偏差，它在跨模态检索任务中的表现不如较小的 CLIP 检索器。为了解决这个问题，我们提出了模态感知的硬负挖掘来减轻 MLLM 检索器表现出的模态偏差。其次，我们建议不断微调通用多模态检索器，以增强其文本检索能力，同时保持多模态检索能力。因此，我们的模型 MM-Embed 在跨多个领域和任务的多模态检索基准 M-BEIR 上实现了最佳性能，同时在 MTEB 检索基准上也超越了最佳文本检索模型 NV-Embed-v1。最后，我们探索使用现成的 MLLM 作为零样本重排器来细化多模态检索器候选者的排名。我们发现，通过提示和重排，当用户查询（例如，文本-图像组合查询）更复杂且更难理解时，MLLM 可以进一步改善多模态检索。这些发现也为未来推进通用多模式检索铺平了道路。

Title: Context-Informed Machine Translation of Manga using Multimodal Large Language Models

• Authors: Philip Lippmann, Konrad Skublicki, Joshua Tanner, Shonosuke Ishiwatari, Jie Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.02589
• Pdf URL: https://arxiv.org/pdf/2411.02589
• Copy Paste: [[2411.02589]] Context-Informed Machine Translation of Manga using Multimodal Large Language Models(https://arxiv.org/abs/2411.02589)
• Keywords: language model, llm
• Abstract: Due to the significant time and effort required for handcrafting translations, most manga never leave the domestic Japanese market. Automatic manga translation is a promising potential solution. However, it is a budding and underdeveloped field and presents complexities even greater than those found in standard translation due to the need to effectively incorporate visual elements into the translation process to resolve ambiguities. In this work, we investigate to what extent multimodal large language models (LLMs) can provide effective manga translation, thereby assisting manga authors and publishers in reaching wider audiences. Specifically, we propose a methodology that leverages the vision component of multimodal LLMs to improve translation quality and evaluate the impact of translation unit size, context length, and propose a token efficient approach for manga translation. Moreover, we introduce a new evaluation dataset -- the first parallel Japanese-Polish manga translation dataset -- as part of a benchmark to be used in future research. Finally, we contribute an open-source software suite, enabling others to benchmark LLMs for manga translation. Our findings demonstrate that our proposed methods achieve state-of-the-art results for Japanese-English translation and set a new standard for Japanese-Polish.
• 摘要：由于手工翻译需要大量的时间和精力，大多数漫画从未离开日本国内市场。自动漫画翻译是一个有前途的潜在解决方案。然而，这是一个新兴且欠发达的领域，由于需要有效地将视觉元素融入翻译过程以解决歧义问题，因此其复杂性甚至高于标准翻译。在这项工作中，我们调查了多模态大型语言模型 (LLM) 在多大程度上可以提供有效的漫画翻译，从而帮助漫画作者和出版商接触更广泛的受众。具体来说，我们提出了一种利用多模态 LLM 的视觉组件来提高翻译质量并评估翻译单元大小、上下文长度影响的方法，并提出了一种高效的漫画翻译方法。此外，我们引入了一个新的评估数据集——第一个并行的日语-波兰语漫画翻译数据集——作为未来研究基准的一部分。最后，我们贡献了一个开源软件套件，使其他人能够对漫画翻译的 LLM 进行基准测试。我们的研究结果表明，我们提出的方法在日语-英语翻译中取得了最先进的成果，并为日语-波兰语树立了新的标准。

Title: FactTest: Factuality Testing in Large Language Models with Statistical Guarantees

• Authors: Fan Nie, Xiaotian Hou, Shuhang Lin, James Zou, Huaxiu Yao, Linjun Zhang
• Subjects: cs.CL, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.02603
• Pdf URL: https://arxiv.org/pdf/2411.02603
• Copy Paste: [[2411.02603]] FactTest: Factuality Testing in Large Language Models with Statistical Guarantees(https://arxiv.org/abs/2411.02603)
• Keywords: language model, llm, hallucination
• Abstract: The propensity of Large Language Models (LLMs) to generate hallucinations and non-factual content undermines their reliability in high-stakes domains, where rigorous control over Type I errors (the conditional probability of incorrectly classifying hallucinations as truthful content) is essential. Despite its importance, formal verification of LLM factuality with such guarantees remains largely unexplored. In this paper, we introduce FactTest, a novel framework that statistically assesses whether an LLM can confidently provide correct answers to given questions with high-probability correctness guarantees. We formulate factuality testing as hypothesis testing problem to enforce an upper bound of Type I errors at user-specified significance levels. Notably, we prove that our framework also ensures strong Type II error control under mild conditions and can be extended to maintain its effectiveness when covariate shifts exist. %These analyses are amenable to the principled NP framework. Our approach is distribution-free and works for any number of human-annotated samples. It is model-agnostic and applies to any black-box or white-box LM. Extensive experiments on question-answering (QA) and multiple-choice benchmarks demonstrate that \approach effectively detects hallucinations and improves the model's ability to abstain from answering unknown questions, leading to an over 40% accuracy improvement.
• 摘要：大型语言模型 (LLM) 倾向于产生幻觉和非事实内容，这削弱了它们在高风险领域的可靠性，在这些领域，对 I 类错误（将幻觉错误地归类为真实内容的条件概率）的严格控制至关重要。尽管它很重要，但对具有此类保证的 LLM 事实性的形式化验证在很大程度上仍未得到探索。在本文中，我们介绍了 FactTest，这是一个新颖的框架，它以统计方式评估 LLM 是否可以自信地为给定的问题提供正确答案，并保证高概率的正确性。我们将事实性测试表述为假设检验问题，以在用户指定的显着性水平上强制执行 I 类错误的上限。值得注意的是，我们证明我们的框架在温和条件下也能确保强大的 II 类错误控制，并且可以扩展以在存在协变量偏移时保持其有效性。%这些分析适用于原则性 NP 框架。我们的方法是无分布的，适用于任意数量的人工注释样本。它与模型无关，适用于任何黑盒或白盒 LM。在问答 (QA) 和多项选择基准上进行的大量实验表明，\approach 可以有效检测幻觉并提高模型避免回答未知问题的能力，从而将准确率提高 40% 以上。

Title: TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network

• Authors: Nouf Alabbasi, Omar Erak, Omar Alhussein, Ismail Lotfi, Sami Muhaidat, Merouane Debbah
• Subjects: cs.CL, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.02617
• Pdf URL: https://arxiv.org/pdf/2411.02617
• Copy Paste: [[2411.02617]] TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network(https://arxiv.org/abs/2411.02617)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: The telecommunications industry's rapid evolution demands intelligent systems capable of managing complex networks and adapting to emerging technologies. While large language models (LLMs) show promise in addressing these challenges, their deployment in telecom environments faces significant constraints due to edge device limitations and inconsistent documentation. To bridge this gap, we present TeleOracle, a telecom-specialized retrieval-augmented generation (RAG) system built on the Phi-2 small language model (SLM). To improve context retrieval, TeleOracle employs a two-stage retriever that incorporates semantic chunking and hybrid keyword and semantic search. Additionally, we expand the context window during inference to enhance the model's performance on open-ended queries. We also employ low-rank adaption for efficient fine-tuning. A thorough analysis of the model's performance indicates that our RAG framework is effective in aligning Phi-2 to the telecom domain in a downstream question and answer (QnA) task, achieving a 30% improvement in accuracy over the base Phi-2 model, reaching an overall accuracy of 81.20%. Notably, we show that our model not only performs on par with the much larger LLMs but also achieves a higher faithfulness score, indicating higher adherence to the retrieved context.
• 摘要：电信行业的快速发展需要能够管理复杂网络并适应新兴技术的智能系统。虽然大型语言模型 (LLM) 在应对这些挑战方面显示出良好的前景，但由于边缘设备限制和文档不一致，它们在电信环境中的部署面临着重大限制。为了弥补这一差距，我们推出了 TeleOracle，这是一种基于 Phi-2 小型语言模型 (SLM) 构建的电信专用检索增强生成 (RAG) 系统。为了改进上下文检索，TeleOracle 采用了两阶段检索器，结合了语义分块和混合关键字和语义搜索。此外，我们在推理过程中扩展了上下文窗口，以提高模型在开放式查询上的性能。我们还采用低秩自适应来实现高效的微调。对模型性能的全面分析表明，我们的 RAG 框架能够有效地将 Phi-2 与下游问答 (QnA) 任务中的电信领域对齐，准确率比基础 Phi-2 模型提高了 30%，总体准确率达到 81.20%。值得注意的是，我们表明我们的模型不仅性能与规模大得多的 LLM 相当，而且还获得了更高的忠实度分数，表明对检索到的上下文的遵循程度更高。

Title: Extracting Unlearned Information from LLMs with Activation Steering

• Authors: Atakan Seyitoğlu, Aleksei Kuvshinov, Leo Schwinn, Stephan Günnemann
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02631
• Pdf URL: https://arxiv.org/pdf/2411.02631
• Copy Paste: [[2411.02631]] Extracting Unlearned Information from LLMs with Activation Steering(https://arxiv.org/abs/2411.02631)
• Keywords: language model, llm
• Abstract: An unintended consequence of the vast pretraining of Large Language Models (LLMs) is the verbatim memorization of fragments of their training data, which may contain sensitive or copyrighted information. In recent years, unlearning has emerged as a solution to effectively remove sensitive knowledge from models after training. Yet, recent work has shown that supposedly deleted information can still be extracted by malicious actors through various attacks. Still, current attacks retrieve sets of possible candidate generations and are unable to pinpoint the output that contains the actual target information. We propose activation steering as a method for exact information retrieval from unlearned LLMs. We introduce a novel approach to generating steering vectors, named Anonymized Activation Steering. Additionally, we develop a simple word frequency method to pinpoint the correct answer among a set of candidates when retrieving unlearned information. Our evaluation across multiple unlearning techniques and datasets demonstrates that activation steering successfully recovers general knowledge (e.g., widely known fictional characters) while revealing limitations in retrieving specific information (e.g., details about non-public individuals). Overall, our results demonstrate that exact information retrieval from unlearned models is possible, highlighting a severe vulnerability of current unlearning techniques.
• 摘要：大型语言模型 (LLM) 的大量预训练的一个意外后果是逐字逐句地记住其训练数据的片段，其中可能包含敏感信息或受版权保护的信息。近年来，反学习已成为一种有效从训练后的模型中删除敏感知识的解决方案。然而，最近的研究表明，恶意行为者仍然可以通过各种攻击提取据称被删除的信息。尽管如此，当前的攻击会检索可能的候选生成集，并且无法精确定位包含实际目标信息的输出。我们提出了一种激活转向方法，用于从未学习的 LLM 中精确检索信息。我们介绍了一种生成转向向量的新方法，称为匿名激活转向。此外，我们还开发了一种简单的词频方法来在检索未学习的信息时在一组候选中精确地找出正确答案。我们对多种反学习技术和数据集的评估表明，激活控制成功恢复了一般知识（例如广为人知的虚构人物），同时揭示了检索特定信息（例如非公开个人的详细信息）的局限性。总体而言，我们的结果表明，从未学习的模型中检索精确信息是可能的，这凸显了当前反学习技术的严重漏洞。

Title: A Comparative Analysis of Counterfactual Explanation Methods for Text Classifiers

• Authors: Stephen McAleese, Mark Keane
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02643
• Pdf URL: https://arxiv.org/pdf/2411.02643
• Copy Paste: [[2411.02643]] A Comparative Analysis of Counterfactual Explanation Methods for Text Classifiers(https://arxiv.org/abs/2411.02643)
• Keywords: language model, llm
• Abstract: Counterfactual explanations can be used to interpret and debug text classifiers by producing minimally altered text inputs that change a classifier's output. In this work, we evaluate five methods for generating counterfactual explanations for a BERT text classifier on two datasets using three evaluation metrics. The results of our experiments suggest that established white-box substitution-based methods are effective at generating valid counterfactuals that change the classifier's output. In contrast, newer methods based on large language models (LLMs) excel at producing natural and linguistically plausible text counterfactuals but often fail to generate valid counterfactuals that alter the classifier's output. Based on these results, we recommend developing new counterfactual explanation methods that combine the strengths of established gradient-based approaches and newer LLM-based techniques to generate high-quality, valid, and plausible text counterfactual explanations.
• 摘要：反事实解释可用于解释和调试文本分类器，方法是生成改变分类器输出的最小更改的文本输入。在这项工作中，我们使用三个评估指标评估了在两个数据集上为 BERT 文本分类器生成反事实解释的五种方法。我们的实验结果表明，成熟的基于白盒替换的方法可以有效地生成改变分类器输出的有效反事实。相比之下，基于大型语言模型 (LLM) 的新方法擅长生成自然且语言上合理的文本反事实，但通常无法生成改变分类器输出的有效反事实。基于这些结果，我们建议开发新的反事实解释方法，结合成熟的基于梯度的方法和较新的基于 LLM 的技术的优势，以生成高质量、有效且合理的文本反事实解释。

Title: Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge

• Authors: Karthik Soman, Andrew Langdon, Catalina Villouta, Chinmay Agrawal, Lashaw Salta, Braian Peetoom, Gianmarco Bellucci, Orion J Buske
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.02657
• Pdf URL: https://arxiv.org/pdf/2411.02657
• Copy Paste: [[2411.02657]] Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge(https://arxiv.org/abs/2411.02657)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: Rare diseases present unique challenges in healthcare, often suffering from delayed diagnosis and fragmented information landscapes. The scarcity of reliable knowledge in these conditions poses a distinct challenge for Large Language Models (LLMs) in supporting clinical management and delivering precise patient information underscoring the need for focused training on these 'zebra' cases. We present Zebra-Llama, a specialized context-aware language model with high precision Retrieval Augmented Generation (RAG) capability, focusing on Ehlers-Danlos Syndrome (EDS) as our case study. EDS, affecting 1 in 5,000 individuals, exemplifies the complexities of rare diseases with its diverse symptoms, multiple subtypes, and evolving diagnostic criteria. By implementing a novel context-aware fine-tuning methodology trained on questions derived from medical literature, patient experiences, and clinical resources, along with expertly curated responses, Zebra-Llama demonstrates unprecedented capabilities in handling EDS-related queries. On a test set of real-world questions collected from EDS patients and clinicians, medical experts evaluated the responses generated by both models, revealing Zebra-Llama's substantial improvements over base model (Llama 3.1-8B-Instruct) in thoroughness (77.5% vs. 70.1%), accuracy (83.0% vs. 78.8%), clarity (74.7% vs. 72.0%) and citation reliability (70.6% vs. 52.3%). Released as an open-source resource, Zebra-Llama not only provides more accessible and reliable EDS information but also establishes a framework for developing specialized AI solutions for other rare conditions. This work represents a crucial step towards democratizing expert-level knowledge in rare disease management, potentially transforming how healthcare providers and patients navigate the complex landscape of rare diseases.
• 摘要：罕见病给医疗保健带来了独特的挑战，通常遭受诊断延迟和信息环境碎片化的影响。在这些情况下，可靠知识的稀缺对大型语言模型 (LLM) 提出了独特的挑战，使其无法支持临床管理和提供准确的患者信息，这凸显了对这些“斑马”病例进行有针对性的训练的必要性。我们介绍了 Zebra-Llama，这是一种具有高精度检索增强生成 (RAG) 功能的专门上下文感知语言模型，以 Ehlers-Danlos 综合征 (EDS) 作为我们的案例研究。EDS 影响 1/5,000 的人，其症状多样、亚型多样、诊断标准不断变化，体现了罕见疾病的复杂性。通过实施一种新颖的上下文感知微调方法，该方法针对来自医学文献、患者体验和临床资源的问题进行训练，并结合专家策划的响应，Zebra-Llama 展示了前所未有的处理 EDS 相关查询的能力。在从 EDS 患者和临床医生那里收集的一组真实问题测试中，医学专家评估了两种模型生成的答案，结果显示 Zebra-Llama 在完整性（77.5% vs. 70.1%）、准确性（83.0% vs. 78.8%）、清晰度（74.7% vs. 72.0%）和引用可靠性（70.6% vs. 52.3%）方面比基础模型 (Llama 3.1-8B-Instruct) 有了显著改进。Zebra-Llama 以开源资源的形式发布，不仅提供了更易于获取和可靠的 EDS 信息，还为开发其他罕见疾病的专门 AI 解决方案建立了框架。这项工作代表着朝着实现罕见疾病管理专家级知识民主化迈出了关键一步，有可能改变医疗保健提供者和患者应对罕见疾病复杂局面的方式。

Title: Wave Network: An Ultra-Small Language Model

• Authors: Xin Zhang, Victor S.Sheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02674
• Pdf URL: https://arxiv.org/pdf/2411.02674
• Copy Paste: [[2411.02674]] Wave Network: An Ultra-Small Language Model(https://arxiv.org/abs/2411.02674)
• Keywords: language model
• Abstract: We propose an innovative token representation and update method in a new ultra-small language model: the Wave network. Specifically, we use a \textbf{complex vector} to represent each token, encoding both global and local semantics of the input text. A \textbf{complex vector} consists of two components: a magnitude vector representing the \textit{global semantics} of the input text, and a phase vector capturing the \textit{relationships between individual tokens and global semantics}. Experiments on the AG News text classification task demonstrate that, when generating complex vectors from randomly initialized token embeddings, our single-layer Wave Network achieves 90.91\% accuracy with wave interference and 91.66\% with wave modulation -- outperforming a single Transformer layer using BERT pre-trained embeddings by 19.23\% and 19.98\%, respectively, and approaching the accuracy of the pre-trained and fine-tuned BERT base model (94.64\%). Additionally, compared to BERT base, the Wave Network reduces video memory usage and training time by 77.34\% and 85.62\% during wave modulation. In summary, we used a 2.4-million-parameter small language model to achieve accuracy comparable to a 100-million-parameter BERT model in text classification.
• 摘要：我们在一种新的超小型语言模型 Wave 网络中提出了一种创新的标记表示和更新方法。具体来说，我们使用 \textbf{复杂向量} 来表示每个标记，对输入文本的全局和局部语义进行编码。\textbf{复杂向量} 由两个部分组成：一个幅度向量，表示输入文本的 \textit{全局语义}，以及一个相位向量，用于捕获 \textit{各个标记与全局语义之间的关系}。在 AG News 文本分类任务上进行的实验表明，当从随机初始化的 token 嵌入生成复杂向量时，我们的单层 Wave 网络在波干扰下可实现 90.91% 的准确率，在波调制下可实现 91.66% 的准确率，比使用 BERT 预训练嵌入的单个 Transformer 层分别高出 19.23% 和 19.98%，接近预训练和微调的 BERT 基础模型的准确率（94.64%）。此外，与 BERT 基础相比，Wave 网络在波调制期间将视频内存使用量和训练时间减少了 77.34% 和 85.62%。总之，我们使用 240 万参数的小型语言模型在文本分类中实现了与 1 亿参数的 BERT 模型相当的准确率。

Title: On the loss of context-awareness in general instruction fine-tuning

• Authors: Yihan Wang, Andrew Bai, Nanyun Peng, Cho-Jui Hsieh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02688
• Pdf URL: https://arxiv.org/pdf/2411.02688
• Copy Paste: [[2411.02688]] On the loss of context-awareness in general instruction fine-tuning(https://arxiv.org/abs/2411.02688)
• Keywords: language model, llm, prompt, chat
• Abstract: Pretrained Large Language Models (LLMs) require post-training methods such as supervised fine-tuning (SFT) on instruction-response pairs to enable instruction following. However, this process can potentially harm existing capabilities learned during pretraining. In this paper, we investigate the loss of context awareness after SFT, defined as the capability to extract and understand information from the user-provided context and respond accordingly. We are the first to identify and show that the loss of context-awareness appears on instruction-finetuned LLMs when the chat template is applied to the input prompts. We identify the performance decline is partially caused by the bias embedded into the chat template to focus less on the user-provided context. Based on these observations, we propose two methods to mitigate the loss of context awareness in instruct models: post-hoc attention steering on user prompts and conditional instruction fine-tuning with a context-dependency indicator. Empirical experiments on 4 context-dependent downstream tasks and 3 pretrained LLMs of different sizes show that our methods effectively mitigates the loss of context awareness without compromising the general ability to follow instructions. Our findings also strongly advocate the necessity to carefully benchmark context awareness after instruction fine-tuning.
• 摘要：预训练大型语言模型 (LLM) 需要后训练方法，例如对指令-响应对进行监督微调 (SFT)，以实现指令跟随。但是，此过程可能会损害预训练期间学到的现有能力。在本文中，我们研究了 SFT 之后上下文感知的丧失，上下文感知的丧失定义为从用户提供的上下文中提取和理解信息并做出相应响应的能力。我们首次发现并表明，当将聊天模板应用于输入提示时，指令微调的 LLM 会出现上下文感知的丧失。我们发现性能下降部分是由于聊天模板中嵌入的偏见导致较少关注用户提供的上下文。基于这些观察，我们提出了两种方法来减轻指令模型中上下文感知的丧失：事后注意引导用户提示和使用上下文依赖性指示器进行条件指令微调。在 4 个上下文相关下游任务和 3 个不同大小的预训练 LLM 上进行的实证实验表明，我们的方法有效地缓解了上下文感知的损失，同时又不损害遵循指令的一般能力。我们的研究结果还强烈主张在指令微调后仔细对上下文感知进行基准测试的必要性。

Title: Game Plot Design with an LLM-powered Assistant: An Empirical Study with Game Designers

• Authors: Seyed Hossein Alavi, Weijia Xu, Nebojsa Jojic, Daniel Kennett, Raymond T. Ng, Sudha Rao, Haiyan Zhang, Bill Dolan, Vered Shwartz
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.02714
• Pdf URL: https://arxiv.org/pdf/2411.02714
• Copy Paste: [[2411.02714]] Game Plot Design with an LLM-powered Assistant: An Empirical Study with Game Designers(https://arxiv.org/abs/2411.02714)
• Keywords: llm
• Abstract: We introduce GamePlot, an LLM-powered assistant that supports game designers in crafting immersive narratives for turn-based games, and allows them to test these games through a collaborative game play and refine the plot throughout the process. Our user study with 14 game designers shows high levels of both satisfaction with the generated game plots and sense of ownership over the narratives, but also reconfirms that LLM are limited in their ability to generate complex and truly innovative content. We also show that diverse user populations have different expectations from AI assistants, and encourage researchers to study how tailoring assistants to diverse user groups could potentially lead to increased job satisfaction and greater creativity and innovation over time.
• 摘要：我们推出了 GamePlot，这是一款由 LLM 提供支持的助手，它支持游戏设计师为回合制游戏制作沉浸式叙事，并允许他们通过协作游戏测试这些游戏，并在整个过程中完善情节。我们对 14 位游戏设计师进行的用户研究表明，他们对生成的游戏情节和叙事都具有很高的满意度，但也再次证实了 LLM 在生成复杂且真正创新的内容方面的能力有限。我们还表明，不同的用户群体对 AI 助手的期望不同，并鼓励研究人员研究如何根据不同的用户群体量身定制助手，这可能会随着时间的推移提高工作满意度并增强创造力和创新能力。

Title: Multimodal Commonsense Knowledge Distillation for Visual Question Answering

• Authors: Shuo Yang, Siwen Luo, Soyeon Caren Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02722
• Pdf URL: https://arxiv.org/pdf/2411.02722
• Copy Paste: [[2411.02722]] Multimodal Commonsense Knowledge Distillation for Visual Question Answering(https://arxiv.org/abs/2411.02722)
• Keywords: language model, llm, prompt
• Abstract: Existing Multimodal Large Language Models (MLLMs) and Visual Language Pretrained Models (VLPMs) have shown remarkable performances in the general Visual Question Answering (VQA). However, these models struggle with VQA questions that require external commonsense knowledge due to the challenges in generating high-quality prompts and the high computational costs of fine-tuning. In this work, we propose a novel graph-based multimodal commonsense knowledge distillation framework that constructs a unified relational graph over commonsense knowledge, visual objects and questions through a Graph Convolutional Network (GCN) following a teacher-student environment. This proposed framework is flexible with any type of teacher and student models without further fine-tuning, and has achieved competitive performances on the ScienceQA dataset.
• 摘要：现有的多模态大型语言模型 (MLLM) 和视觉语言预训练模型 (VLPM) 在通用视觉问答 (VQA) 中表现出色。然而，由于生成高质量提示的挑战和微调的高计算成本，这些模型在处理需要外部常识知识的 VQA 问题时遇到了困难。在这项工作中，我们提出了一种新颖的基于图的多模态常识知识提炼框架，该框架通过遵循师生环境的图卷积网络 (GCN) 构建常识知识、视觉对象和问题的统一关系图。该提出的框架可灵活适用于任何类型的教师和学生模型，无需进一步微调，并且在 ScienceQA 数据集上取得了具有竞争力的表现。

Title: A Natural Language Processing Approach to Support Biomedical Data Harmonization: Leveraging Large Language Models

• Authors: Zexu Li, Suraj P. Prabhu, Zachary T. Popp, Shubhi S. Jain, Vijetha Balakundi, Ting Fang Alvin Ang, Rhoda Au, Jinying Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02730
• Pdf URL: https://arxiv.org/pdf/2411.02730
• Copy Paste: [[2411.02730]] A Natural Language Processing Approach to Support Biomedical Data Harmonization: Leveraging Large Language Models(https://arxiv.org/abs/2411.02730)
• Keywords: language model, llm
• Abstract: Biomedical research requires large, diverse samples to produce unbiased results. Automated methods for matching variables across datasets can accelerate this process. Research in this area has been limited, primarily focusing on lexical matching and ontology based semantic matching. We aimed to develop new methods, leveraging large language models (LLM) and ensemble learning, to automate variable matching. Methods: We utilized data from two GERAS cohort (European and Japan) studies to develop variable matching methods. We first manually created a dataset by matching 352 EU variables with 1322 candidate JP variables, where matched variable pairs were positive and unmatched pairs were negative instances. Using this dataset, we developed and evaluated two types of natural language processing (NLP) methods, which matched variables based on variable labels and definitions from data dictionaries: (1) LLM-based and (2) fuzzy matching. We then developed an ensemble-learning method, using the Random Forest model, to integrate individual NLP methods. RF was trained and evaluated on 50 trials. Each trial had a random split (4:1) of training and test sets, with the model's hyperparameters optimized through cross-validation on the training set. For each EU variable, 1322 candidate JP variables were ranked based on NLP-derived similarity scores or RF's probability scores, denoting their likelihood to match the EU variable. Ranking performance was measured by top-n hit ratio (HRn) and mean reciprocal rank (MRR). Results:E5 performed best among individual methods, achieving 0.90 HR-30 and 0.70 MRR. RF performed better than E5 on all metrics over 50 trials (P less than 0.001) and achieved an average HR 30 of 0.98 and MRR of 0.73. LLM-derived features contributed most to RF's performance. One major cause of errors in automatic variable matching was ambiguous variable definitions within data dictionaries.
• 摘要：生物医学研究需要大量、多样化的样本才能产生无偏结果。跨数据集匹配变量的自动化方法可以加速这一过程。该领域的研究有限，主要集中在词汇匹配和基于本体的语义匹配上。我们旨在开发新方法，利用大型语言模型 (LLM) 和集成学习来自动化变量匹配。方法：我们利用来自两个 GERAS 队列（欧洲和日本）研究的数据来开发变量匹配方法。我们首先手动创建一个数据集，将 352 个 EU 变量与 1322 个候选 JP 变量进行匹配，其中匹配的变量对为正，不匹配的变量对为负。使用此数据集，我们开发并评估了两种类型的自然语言处理 (NLP) 方法，它们根据数据字典中的变量标签和定义匹配变量：(1) 基于 LLM 和 (2) 模糊匹配。然后，我们使用随机森林模型开发了一种集成学习方法来集成单个 NLP 方法。RF 在 50 次试验中进行了训练和评估。每次试验都会随机（4:1）分割训练集和测试集，并通过在训练集上进行交叉验证来优化模型的超参数。对于每个 EU 变量，1322 个候选 JP 变量根据 NLP 得出的​​相似度得分或 RF 的概率得分进行排名，表示它们与 EU 变量匹配的可能性。排名性能通过前 n 个命中率 (HRn) 和平均倒数排名 (MRR) 来衡量。结果：E5 在各个方法中表现最佳，达到 0.90 HR-30 和 0.70 MRR。在 50 次试验中，RF 在所有指标上的表现均优于 E5（P 小于 0.001），平均 HR 30 为 0.98，MRR 为 0.73。LLM 得出的特征对 RF 的性能贡献最大。自动变量匹配错误的主要原因之一是数据字典中的变量定义不明确。

Title: Novelty-focused R&D landscaping using transformer and local outlier factor

• Authors: Jaewoong Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.02738
• Pdf URL: https://arxiv.org/pdf/2411.02738
• Copy Paste: [[2411.02738]] Novelty-focused R&D landscaping using transformer and local outlier factor(https://arxiv.org/abs/2411.02738)
• Keywords: language model
• Abstract: While numerous studies have explored the field of research and development (R&D) landscaping, the preponderance of these investigations has emphasized predictive analysis based on R&D outcomes, specifically patents, and academic literature. However, the value of research proposals and novelty analysis has seldom been addressed. This study proposes a systematic approach to constructing and navigating the R&D landscape that can be utilized to guide organizations to respond in a reproducible and timely manner to the challenges presented by increasing number of research proposals. At the heart of the proposed approach is the composite use of the transformer-based language model and the local outlier factor (LOF). The semantic meaning of the research proposals is captured with our further-trained transformers, thereby constructing a comprehensive R&D landscape. Subsequently, the novelty of the newly selected research proposals within the annual landscape is quantified on a numerical scale utilizing the LOF by assessing the dissimilarity of each proposal to others preceding and within the same year. A case study examining research proposals in the energy and resource sector in South Korea is presented. The systematic process and quantitative outcomes are expected to be useful decision-support tools, providing future insights regarding R&D planning and roadmapping.
• 摘要：虽然已经有许多研究探索了研发 (R&D) 格局，但这些研究主要强调基于研发成果（特别是专利和学术文献）的预测分析。然而，研究提案和新颖性分析的价值很少得到重视。本研究提出了一种构建和导航研发格局的系统方法，可用于指导组织以可重复和及时的方式应对越来越多的研究提案所带来的挑战。所提出方法的核心是基于转换器的语言模型和局部异常因子 (LOF) 的综合使用。我们通过进一步训练的转换器捕捉研究提案的语义含义，从而构建全面的研发格局。随后，利用 LOF 通过评估每个提案与同一年之前和同一年内其他提案的差异，以数值尺度量化年度格局中新选定的研究提案的新颖性。本文介绍了韩国能源和资源领域研究提案的案例研究。系统流程和定量结果有望成为有用的决策支持工具，为未来研发规划和路线图提供见解。

Title: Language Models and Cycle Consistency for Self-Reflective Machine Translation

• Authors: Jianqiao Wangni
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG, cs.NE, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.02791
• Pdf URL: https://arxiv.org/pdf/2411.02791
• Copy Paste: [[2411.02791]] Language Models and Cycle Consistency for Self-Reflective Machine Translation(https://arxiv.org/abs/2411.02791)
• Keywords: language model, llm
• Abstract: This paper introduces a novel framework that leverages large language models (LLMs) for machine translation (MT). We start with one conjecture: an ideal translation should contain complete and accurate information for a strong enough LLM to recover the original sentence. We generate multiple translation candidates from a source language A to a target language B, and subsequently translate these candidates back to the original language A. By evaluating the cycle consistency between the original and back-translated sentences using metrics such as token-level precision and accuracy, we implicitly estimate the translation quality in language B, without knowing its ground-truth. This also helps to evaluate the LLM translation capability, only with monolingual corpora. For each source sentence, we identify the translation candidate with optimal cycle consistency with the original sentence as the final answer. Our experiments demonstrate that larger LLMs, or the same LLM with more forward passes during inference, exhibit increased cycle consistency, aligning with the LLM model size scaling law and test-time computation scaling law. This work provide methods for, 1) to implicitly evaluate translation quality of a sentence in the target language, 2), to evaluate capability of LLM for any-to-any-language translation, and 3), how to generate a better translation for a specific LLM.
• 摘要：本文介绍了一种利用大型语言模型 (LLM) 进行机器翻译 (MT) 的新框架。我们从一个猜想开始：理想的翻译应该包含完整而准确的信息，以便足够强大的 LLM 能够恢复原始句子。我们生成从源语言 A 到目标语言 B 的多个翻译候选，然后将这些候选翻译回原始语言 A。通过使用标记级精度和准确度等指标评估原始句子和回译句子之间的循环一致性，我们可以隐式估计语言 B 的翻译质量，而无需了解其基本事实。这也有助于仅使用单语语料库来评估 LLM 的翻译能力。对于每个源句子，我们确定与原始句子具有最佳循环一致性的翻译候选作为最终答案。我们的实验表明，更大的 LLM 或在推理过程中具有更多前向传递的相同 LLM 表现出更高的循环一致性，这与 LLM 模型大小缩放定律和测试时间计算缩放定律一致。这项工作提供了以下方法：1）隐性评估目标语言句子的翻译质量；2）评估 LLM 对任何语言的翻译能力；3）如何为特定的 LLM 生成更好的翻译。

Title: The Evolution of RWKV: Advancements in Efficient Language Modeling

• Authors: Akul Datta
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02795
• Pdf URL: https://arxiv.org/pdf/2411.02795
• Copy Paste: [[2411.02795]] The Evolution of RWKV: Advancements in Efficient Language Modeling(https://arxiv.org/abs/2411.02795)
• Keywords: language model
• Abstract: This paper reviews the development of the Receptance Weighted Key Value (RWKV) architecture, emphasizing its advancements in efficient language modeling. RWKV combines the training efficiency of Transformers with the inference efficiency of RNNs through a novel linear attention mechanism. We examine its core innovations, adaptations across various domains, and performance advantages over traditional models. The paper also discusses challenges and future directions for RWKV as a versatile architecture in deep learning.
• 摘要：本文回顾了接受加权键值 (RWKV) 架构的发展，强调了其在高效语言建模方面的进步。RWKV 通过一种新颖的线性注意机制将 Transformers 的训练效率与 RNN 的推理效率相结合。我们研究了它的核心创新、跨领域的适应性以及相对于传统模型的性能优势。本文还讨论了 RWKV 作为深度学习中的多功能架构所面临的挑战和未来方向。

Title: Mixtures of In-Context Learners

• Authors: Giwon Hong, Emile van Krieken, Edoardo Ponti, Nikolay Malkin, Pasquale Minervini
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02830
• Pdf URL: https://arxiv.org/pdf/2411.02830
• Copy Paste: [[2411.02830]] Mixtures of In-Context Learners(https://arxiv.org/abs/2411.02830)
• Keywords: llm
• Abstract: In-context learning (ICL) adapts LLMs by providing demonstrations without fine-tuning the model parameters; however, it does not differentiate between demonstrations and quadratically increases the complexity of Transformer LLMs, exhausting the memory. As a solution, we propose Mixtures of In-Context Learners (MoICL), a novel approach to treat subsets of demonstrations as experts and learn a weighting function to merge their output distributions based on a training set. In our experiments, we show performance improvements on 5 out of 7 classification datasets compared to a set of strong baselines (up to +13\% compared to ICL and LENS). Moreover, we enhance the Pareto frontier of ICL by reducing the inference time needed to achieve the same performance with fewer demonstrations. Finally, MoICL is more robust to out-of-domain (up to +11\%), imbalanced (up to +49\%), or noisy demonstrations (up to +38\%) or can filter these out from datasets. Overall, MoICL is a more expressive approach to learning from demonstrations without exhausting the context window or memory.
• 摘要：上下文学习 (ICL) 通过提供演示而不对模型参数进行微调来调整 LLM；然而，它不区分演示，并且二次增加了 Transformer LLM 的复杂性，从而耗尽内存。作为一种解决方案，我们提出了上下文学习器混合 (MoICL)，这是一种新颖的方法，将演示子集视为专家，并学习加权函数以基于训练集合并它们的输出分布。在我们的实验中，与一组强大的基线相比，我们在 7 个分类数据集中的 5 个上展示了性能改进（与 ICL 和 LENS 相比高达 +13\%）。此外，我们通过减少以更少的演示实现相同性能所需的推理时间来增强 ICL 的帕累托前沿。最后，MoICL 对域外（高达 +11\%）、不平衡（高达 +49\%）或嘈杂的演示（高达 +38\%）更稳健，或者可以从数据集中过滤掉这些。总体而言，MoICL 是一种更具表现力的从演示中学习的方法，且不会耗尽上下文窗口或记忆。

Title: PersianRAG: A Retrieval-Augmented Generation System for Persian Language

• Authors: Hossein Hosseini, Mohammad Siobhan Zare, Amir Hossein Mohammadi, Arefeh Kazemi, Zahra Zojaji, Mohammad Ali Nematbakhsh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02832
• Pdf URL: https://arxiv.org/pdf/2411.02832
• Copy Paste: [[2411.02832]] PersianRAG: A Retrieval-Augmented Generation System for Persian Language(https://arxiv.org/abs/2411.02832)
• Keywords: language model, prompt, retrieval augmented generation, retrieval-augmented generation
• Abstract: Retrieval augmented generation (RAG) models, which integrate large-scale pre-trained generative models with external retrieval mechanisms, have shown significant success in various natural language processing (NLP) tasks. However, applying RAG models in Persian language as a low-resource language, poses distinct challenges. These challenges primarily involve the preprocessing, embedding, retrieval, prompt construction, language modeling, and response evaluation of the system. In this paper, we address the challenges towards implementing a real-world RAG system for Persian language called PersianRAG. We propose novel solutions to overcome these obstacles and evaluate our approach using several Persian benchmark datasets. Our experimental results demonstrate the capability of the PersianRAG framework to enhance question answering task in Persian.
• 摘要：检索增强生成 (RAG) 模型将大规模预训练生成模型与外部检索机制相结合，已在各种自然语言处理 (NLP) 任务中取得了显著成功。然而，将 RAG 模型应用于资源匮乏的波斯语语言中，面临着独特的挑战。这些挑战主要涉及系统的预处理、嵌入、检索、提示构建、语言建模和响应评估。在本文中，我们解决了为波斯语语言实现真实 RAG 系统（称为 PersianRAG）所面临的挑战。我们提出了克服这些障碍的新解决方案，并使用多个波斯语基准数据集评估了我们的方法。我们的实验结果证明了 PersianRAG 框架能够增强波斯语问答任务的能力。

Title: Graph-DPEP: Decomposed Plug and Ensemble Play for Few-Shot Document Relation Extraction with Graph-of-Thoughts Reasoning

• Authors: Tao Zhang, Ning Yan, Masood Mortazavi, Hoang H. Nguyen, Zhongfen Deng, Philip S. Yu
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.02864
• Pdf URL: https://arxiv.org/pdf/2411.02864
• Copy Paste: [[2411.02864]] Graph-DPEP: Decomposed Plug and Ensemble Play for Few-Shot Document Relation Extraction with Graph-of-Thoughts Reasoning(https://arxiv.org/abs/2411.02864)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) pre-trained on massive corpora have demonstrated impressive few-shot learning capability on many NLP tasks. Recasting an NLP task into a text-to-text generation task is a common practice so that generative LLMs can be prompted to resolve it. However, performing document-level relation extraction (DocRE) tasks with generative LLM models is still challenging due to the structured output format of DocRE, which complicates the conversion to plain text. Limited information available in few-shot samples and prompt instructions induce further difficulties and challenges in relation extraction for mentioned entities in a document. In this paper, we represent the structured output as a graph-style triplet rather than natural language expressions and leverage generative LLMs for the DocRE task. Our approach, the Graph-DPEP framework is grounded in the reasoning behind triplet explanation thoughts presented in natural language. In this framework, we first introduce a ``decomposed-plug" method for performing the generation from LLMs over prompts with type-space decomposition to alleviate the burden of distinguishing all relation types. Second, we employ a verifier for calibrating the generation and identifying overlooked query entity pairs. Third, we develop "ensemble-play", reapplying generation on the entire type list by leveraging the reasoning thoughts embedded in a sub-graph associated with the missing query pair to address the missingness issue. Through extensive comparisons with existing prompt techniques and alternative Language Models (LLMs), our framework demonstrates superior performance on publicly available benchmarks in experiments.
• 摘要：在海量语料库上预训练的大型语言模型 (LLM) 在许多 NLP 任务上展示了令人印象深刻的少样本学习能力。将 NLP 任务重塑为文本到文本生成任务是一种常见做法，以便可以提示生成式 LLM 来解决它。但是，由于 DocRE 的结构化输出格式使转换为纯文本变得复杂，因此使用生成式 LLM 模型执行文档级关系提取 (DocRE) 任务仍然具有挑战性。少样本中可用的信息有限以及提示指令给文档中提及实体的关系提取带来了进一步的困难和挑战。在本文中，我们将结构化输出表示为图形样式的三元组而不是自然语言表达式，并利用生成式 LLM 完成 DocRE 任务。我们的方法，即 Graph-DPEP 框架，以自然语言中呈现的三元组解释思想背后的推理为基础。在这个框架中，我们首先引入了一种“分解插件”方法，用于通过类型空间分解执行 LLM 生成，以减轻区分所有关系类型的负担。其次，我们使用验证器来校准生成并识别被忽略的查询实体对。第三，我们开发了“集成游戏”，通过利用与缺失查询对相关的子图中嵌入的推理思想来解决缺失问题，在整个类型列表上重新应用生成。通过与现有提示技术和替代语言模型 (LLM) 进行广泛的比较，我们的框架在实验中展示了公开基准测试的卓越性能。

Title: TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection

• Authors: Wei Wu, Zhuoshi Pan, Chao Wang, Liyi Chen, Yunchu Bai, Kun Fu, Zheng Wang, Hui Xiong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02886
• Pdf URL: https://arxiv.org/pdf/2411.02886
• Copy Paste: [[2411.02886]] TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection(https://arxiv.org/abs/2411.02886)
• Keywords: language model, llm
• Abstract: With the development of large language models (LLMs), the ability to handle longer contexts has become a key capability for Web applications such as cross-document understanding and LLM-powered search systems. However, this progress faces two major challenges: performance degradation due to sequence lengths out-of-distribution, and excessively long inference times caused by the quadratic computational complexity of attention. These issues hinder the application of LLMs in long-context scenarios. In this paper, we propose Dynamic Token-Level KV Cache Selection (TokenSelect), a model-agnostic, training-free method for efficient and accurate long-context inference. TokenSelect builds upon the observation of non-contiguous attention sparsity, using Query-Key dot products to measure per-head KV Cache criticality at token-level. By per-head soft voting mechanism, TokenSelect selectively involves a small number of critical KV cache tokens in the attention calculation without sacrificing accuracy. To further accelerate TokenSelect, we designed the Selection Cache based on observations of consecutive Query similarity and implemented efficient dot product kernel, significantly reducing the overhead of token selection. A comprehensive evaluation of TokenSelect demonstrates up to 23.84x speedup in attention computation and up to 2.28x acceleration in end-to-end latency, while providing superior performance compared to state-of-the-art long-context inference methods.
• 摘要：随着大型语言模型 (LLM) 的发展，处理较长上下文的能力已成为跨文档理解和基于 LLM 的搜索系统等 Web 应用的关键能力。然而，这一进步面临两大挑战：由于序列长度不均匀导致的性能下降，以及注意力机制的二次计算复杂度导致的推理时间过长。这些问题阻碍了 LLM 在长上下文场景中的应用。在本文中，我们提出了动态 Token 级 KV 缓存选择 (TokenSelect)，一种与模型无关、无需训练的方法，用于高效、准确的长上下文推理。TokenSelect 建立在对非连续注意力稀疏性的观察之上，使用查询键点积来测量 token 级别的每人 KV 缓存关键性。通过每人软投票机制，TokenSelect 有选择地将少量关键 KV 缓存 token 纳入注意力计算，而不会牺牲准确性。为了进一步加速 TokenSelect，我们根据连续查询相似度的观察设计了 Selection Cache，并实现了高效的点积核，从而显著降低了 token 选择的开销。对 TokenSelect 的全面评估表明，注意力计算速度提高了 23.84 倍，端到端延迟提高了 2.28 倍，同时与最先进的长上下文推理方法相比提供了卓越的性能。

Title: Textual Aesthetics in Large Language Models

• Authors: Lingjie Jiang, Shaohan Huang, Xun Wu, Furu Wei
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02930
• Pdf URL: https://arxiv.org/pdf/2411.02930
• Copy Paste: [[2411.02930]] Textual Aesthetics in Large Language Models(https://arxiv.org/abs/2411.02930)
• Keywords: language model, llm
• Abstract: Image aesthetics is a crucial metric in the field of image generation. However, textual aesthetics has not been sufficiently explored. With the widespread application of large language models (LLMs), previous work has primarily focused on the correctness of content and the helpfulness of responses. Nonetheless, providing responses with textual aesthetics is also an important factor for LLMs, which can offer a cleaner layout and ensure greater consistency and coherence in content. In this work, we introduce a pipeline for aesthetics polishing and help construct a textual aesthetics dataset named TexAes. We propose a textual aesthetics-powered fine-tuning method based on direct preference optimization, termed TAPO, which leverages textual aesthetics without compromising content correctness. Additionally, we develop two evaluation methods for textual aesthetics based on text and image analysis, respectively. Our experiments demonstrate that using textual aesthetics data and employing the TAPO fine-tuning method not only improves aesthetic scores but also enhances performance on general evaluation datasets such as AlpacalEval and Anera-hard.
• 摘要：图像美学是图像生成领域的一个重要指标。然而，文本美学尚未得到充分探索。随着大型语言模型 (LLM) 的广泛应用，以前的工作主要集中在内容的正确性和响应的有用性上。尽管如此，提供具有文本美学的响应也是 LLM 的一个重要因素，它可以提供更清晰的布局并确保内容的一致性和连贯性。在这项工作中，我们引入了一个美学打磨流程，并帮助构建了一个名为 TexAes 的文本美学数据集。我们提出了一种基于直接偏好优化的文本美学驱动的微调方法，称为 TAPO，它在不影响内容正确性的情况下利用文本美学。此外，我们分别基于文本和图像分析开发了两种文本美学评估方法。我们的实验表明，使用文本美学数据并采用 TAPO 微调方法不仅可以提高美学分数，还可以提高在 AlpacalEval 和 Anera-hard 等通用评估数据集上的表现。

Title: Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent

• Authors: Yangning Li, Yinghui Li, Xingyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai-Tao Zheng, Philip S. Yu, Fei Huang, Jingren Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.02937
• Pdf URL: https://arxiv.org/pdf/2411.02937
• Copy Paste: [[2411.02937]] Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent(https://arxiv.org/abs/2411.02937)
• Keywords: language model, llm, hallucination, retrieval augmented generation, agent
• Abstract: Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. The code and dataset will be open-sourced at this https URL.
• 摘要：多模态检索增强生成 (mRAG) 在缓解多模态大型语言模型 (MLLM) 固有的“幻觉”问题方面发挥着重要作用。尽管前景光明，但现有的启发式 mRAG 通常预定义固定的检索过程，这会导致两个问题：(1) 非自适应检索查询。(2) 检索查询过载。然而，这些缺陷无法在当前寻求知识的视觉问答 (VQA) 数据集中得到充分反映，因为最需要的知识可以通过标准的两步检索轻松获得。为了弥补数据集差距，我们首先构建了 Dyn-VQA 数据集，它由三种类型的“动态”问题组成，这些问题需要复杂的知识检索策略，这些策略在查询、工具和时间上各不相同：(1) 答案快速变化的问题。(2) 需要多模态知识的问题。(3) 多跳问题。 Dyn-VQA 上的实验表明，由于现有的启发式 mRAG 的检索过程比较僵化，因此很难为动态问题提供足够且精确的相关知识。因此，我们进一步提出了第一个用于多模态检索的自适应规划代理 OmniSearch。其基本思想是模拟问题解决方案中的人类行为，将复杂的多模态问题动态分解为具有检索动作的子问题链。大量实验证明了我们的 OmniSearch 的有效性，也为推进 mRAG 提供了方向。代码和数据集将在此 https URL 上开源。

Title: A Post-Training Enhanced Optimization Approach for Small Language Models

• Authors: Keke Zhai
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.02939
• Pdf URL: https://arxiv.org/pdf/2411.02939
• Copy Paste: [[2411.02939]] A Post-Training Enhanced Optimization Approach for Small Language Models(https://arxiv.org/abs/2411.02939)
• Keywords: language model
• Abstract: This paper delves into the continuous post-training optimization methods for small language models, and proposes a continuous post-training alignment data construction method for small language models. The core of this method is based on the data guidance of large models, optimizing the diversity and accuracy of alignment data. In addition, to verify the effectiveness of the methods in this paper, we used Qwen2-0.5B-Instruct model as the baseline model for small language models, using the alignment dataset constructed by our proposed method, we trained and compared several groups of experiments, including SFT (Supervised Fine Tuning) post-training experiment and KTO (Kahneman Tversky optimization) post-training experiment, as well as SFT-KTO two-stage post-training experiment and model weight fusion experiment. Finally, we evaluated and analyzed the performance of post-training models, and confirmed that the continuous post-training optimization method proposed by us can significantly improve the performance of small language models.
• 摘要：本文深入研究了针对小语言模型的持续训练后优化方法，提出了一种针对小语言模型的持续训练后对齐数据构建方法。该方法的核心是基于大模型的数据指导，优化对齐数据的多样性和准确性。此外，为了验证本文方法的有效性，以Qwen2-0.5B-Instruct模型作为小语言模型的基线模型，利用本文方法构建的对齐数据集，训练并对比了多组实验，包括SFT（Supervised Fine Tuning）后训练实验和KTO（Kahneman Tversky优化）后训练实验，以及SFT-KTO两阶段后训练实验和模型权重融合实验。最后对训练后模型的性能进行了评估和分析，证实了本文提出的持续训练后优化方法可以显著提升小语言模型的性能。

Title: Capturing research literature attitude towards Sustainable Development Goals: an LLM-based topic modeling approach

• Authors: Francesco Invernici, Francesca Curati, Jelena Jakimov, Amirhossein Samavi, Anna Bernasconi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.02943
• Pdf URL: https://arxiv.org/pdf/2411.02943
• Copy Paste: [[2411.02943]] Capturing research literature attitude towards Sustainable Development Goals: an LLM-based topic modeling approach(https://arxiv.org/abs/2411.02943)
• Keywords: llm
• Abstract: The world is facing a multitude of challenges that hinder the development of human civilization and the well-being of humanity on the planet. The Sustainable Development Goals (SDGs) were formulated by the United Nations in 2015 to address these global challenges by 2030. Natural language processing techniques can help uncover discussions on SDGs within research literature. We propose a completely automated pipeline to 1) fetch content from the Scopus database and prepare datasets dedicated to five groups of SDGs; 2) perform topic modeling, a statistical technique used to identify topics in large collections of textual data; and 3) enable topic exploration through keywords-based search and topic frequency time series extraction. For topic modeling, we leverage the stack of BERTopic scaled up to be applied on large corpora of textual documents (we find hundreds of topics on hundreds of thousands of documents), introducing i) a novel LLM-based embeddings computation for representing scientific abstracts in the continuous space and ii) a hyperparameter optimizer to efficiently find the best configuration for any new big datasets. We additionally produce the visualization of results on interactive dashboards reporting topics' temporal evolution. Results are made inspectable and explorable, contributing to the interpretability of the topic modeling process. Our proposed LLM-based topic modeling pipeline for big-text datasets allows users to capture insights on the evolution of the attitude toward SDGs within scientific abstracts in the 2006-2023 time span. All the results are reproducible by using our system; the workflow can be generalized to be applied at any point in time to any big corpus of textual documents.
• 摘要：世界正面临着众多挑战，阻碍着人类文明的发展和地球上人类的福祉。联合国于 2015 年制定了可持续发展目标 (SDG)，以在 2030 年前解决这些全球挑战。自然语言处理技术可以帮助在研究文献中发现有关 SDG 的讨论。我们提出了一个完全自动化的流程来 1) 从 Scopus 数据库中获取内容并准备专用于五组 SDG 的数据集；2) 执行主题建模，这是一种用于识别大量文本数据中的主题的统计技术；3) 通过基于关键字的搜索和主题频率时间序列提取实现主题探索。对于主题建模，我们利用 BERTopic 堆栈扩展以应用于大量文本文档（我们在数十万份文档中找到数百个主题），引入 i) 一种新颖的基于 LLM 的嵌入计算，用于在连续空间中表示科学摘要和 ii) 超参数优化器，以有效地为任何新的大数据集找到最佳配置。我们还在交互式仪表板上对结果进行了可视化，报告了主题的时间演变。结果变得可检查和可探索，有助于提高主题建模过程的可解释性。我们提出的基于 LLM 的大型文本数据集主题建模流程允许用户在 2006-2023 年的时间跨度内洞察科学摘要中对 SDG 态度的演变。使用我们的系统可以重现所有结果；该工作流程可以推广到任何时间点，应用于任何大型文本文档集。

Title: [Vision Paper] PRObot: Enhancing Patient-Reported Outcome Measures for Diabetic Retinopathy using Chatbots and Generative AI

• Authors: Maren Pielka, Tobias Schneider, Jan Terheyden, Rafet Sifa
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.02973
• Pdf URL: https://arxiv.org/pdf/2411.02973
• Copy Paste: [[2411.02973]] [Vision Paper] PRObot: Enhancing Patient-Reported Outcome Measures for Diabetic Retinopathy using Chatbots and Generative AI(https://arxiv.org/abs/2411.02973)
• Keywords: language model, llm, chat
• Abstract: We present an outline of the first large language model (LLM) based chatbot application in the context of patient-reported outcome measures (PROMs) for diabetic retinopathy. By utilizing the capabilities of current LLMs, we enable patients to provide feedback about their quality of life and treatment progress via an interactive application. The proposed framework offers significant advantages over the current approach, which encompasses only qualitative collection of survey data or a static survey with limited answer options. Using the PROBot LLM-PROM application, patients will be asked tailored questions about their individual challenges, and can give more detailed feedback on the progress of their treatment. Based on this input, we will use machine learning to infer conventional PROM scores, which can be used by clinicians to evaluate the treatment status. The goal of the application is to improve adherence to the healthcare system and treatments, and thus ultimately reduce cases of subsequent vision impairment. The approach needs to be further validated using a survey and a clinical study.
• 摘要：我们概述了第一个基于大型语言模型 (LLM) 的聊天机器人应用程序，该应用程序针对糖尿病视网膜病变患者报告结果测量 (PROM)。通过利用当前 LLM 的功能，我们使患者能够通过交互式应用程序提供有关其生活质量和治疗进展的反馈。与当前方法相比，所提出的框架具有显著优势，当前方法仅包含定性调查数据收集或答案选项有限的静态调查。使用 PROBot LLM-PROM 应用程序，患者将被问及有关其个人挑战的定制问题，并可以就其治疗进展提供更详细的反馈。基于此输入，我们将使用机器学习推断出传统的 PROM 分数，临床医生可以使用该分数来评估治疗状态。该应用程序的目标是提高对医疗保健系统和治疗的依从性，从而最终减少随后视力受损的病例。该方法需要通过调查和临床研究进一步验证。

Title: Growing a Tail: Increasing Output Diversity in Large Language Models

• Authors: Michal Shur-Ofry, Bar Horowitz-Amsalem, Adir Rahamim, Yonatan Belinkov
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.02989
• Pdf URL: https://arxiv.org/pdf/2411.02989
• Copy Paste: [[2411.02989]] Growing a Tail: Increasing Output Diversity in Large Language Models(https://arxiv.org/abs/2411.02989)
• Keywords: language model, prompt
• Abstract: How diverse are the outputs of large language models when diversity is desired? We examine the diversity of responses of various models to questions with multiple possible answers, comparing them with human responses. Our findings suggest that models' outputs are highly concentrated, reflecting a narrow, mainstream 'worldview', in comparison to humans, whose responses exhibit a much longer-tail. We examine three ways to increase models' output diversity: 1) increasing generation randomness via temperature sampling; 2) prompting models to answer from diverse perspectives; 3) aggregating outputs from several models. A combination of these measures significantly increases models' output diversity, reaching that of humans. We discuss implications of these findings for AI policy that wishes to preserve cultural diversity, an essential building block of a democratic social fabric.
• 摘要：当需要多样性时，大型语言模型的输出有多多样化？我们研究了各种模型对具有多个可能答案的问题的响应多样性，并将其与人类的响应进行比较。我们的研究结果表明，与人类相比，模型的输出高度集中，反映了一种狭隘的主流“世界观”，而人类的响应则表现出更长的尾部。我们研究了三种增加模型输出多样性的方法：1）通过温度采样增加生成随机性；2）促使模型从不同角度回答；3）聚合来自多个模型的输出。这些措施的组合显着增加了模型的输出多样性，达到人类的水平。我们讨论了这些发现对希望保护文化多样性的人工智能政策的影响，文化多样性是民主社会结构的基本组成部分。

Title: Leveraging Large Language Models in Code Question Answering: Baselines and Issues

• Authors: Georgy Andryushchenko, Vladimir Ivanov, Vladimir Makharev, Elizaveta Tukhtina, Aidar Valeev
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03012
• Pdf URL: https://arxiv.org/pdf/2411.03012
• Copy Paste: [[2411.03012]] Leveraging Large Language Models in Code Question Answering: Baselines and Issues(https://arxiv.org/abs/2411.03012)
• Keywords: language model
• Abstract: Question answering over source code provides software engineers and project managers with helpful information about the implemented features of a software product. This paper presents a work devoted to using large language models for question answering over source code in Python. The proposed method for implementing a source code question answering system involves fine-tuning a large language model on a unified dataset of questions and answers for Python code. To achieve the highest quality answers, we tested various models trained on datasets preprocessed in different ways: a dataset without grammar correction, a dataset with grammar correction, and a dataset augmented with the generated summaries. The model answers were also analyzed for errors manually. We report BLEU-4, BERTScore F1, BLEURT, and Exact Match metric values, along with the conclusions from the manual error analysis. The obtained experimental results highlight the current problems of the research area, such as poor quality of the public genuine question-answering datasets. In addition, the findings include the positive effect of the grammar correction of the training data on the testing metric values. The addressed findings and issues could be important for other researchers who attempt to improve the quality of source code question answering solutions. The training and evaluation code is publicly available at this https URL.
• 摘要：通过源代码进行问答可以为软件工程师和项目经理提供有关软件产品已实现功能的有用信息。本文介绍了一项致力于使用大型语言模型在 Python 源代码上进行问答的工作。所提出的实现源代码问答系统的方法涉及在 Python 代码的统一问题和答案数据集上微调大型语言模型。为了获得最高质量的答案，我们测试了在以不同方式预处理的数据集上训练的各种模型：没有语法校正的数据集、有语法校正的数据集和使用生成的摘要增强的数据集。还手动分析了模型答案中的错误。我们报告了 BLEU-4、BERTScore F1、BLEURT 和 Exact Match 度量值，以及手动错误分析的结论。获得的实验结果突出了该研究领域当前的问题，例如公共真实问答数据集的质量差。此外，研究结果还包括训练数据的语法校正对测试度量值的积极影响。所讨论的发现和问题对于试图提高源代码问答解决方案质量的其他研究人员可能很重要。训练和评估代码在此 https URL 上公开提供。

Title: Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning

• Authors: Bei Li, Tong Zheng, Rui Wang, Jiahao Liu, Qingyan Guo, Junliang Guo, Xu Tan, Tong Xiao, Jingbo Zhu, Jingang Wang, Xunliang Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03042
• Pdf URL: https://arxiv.org/pdf/2411.03042
• Copy Paste: [[2411.03042]] Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning(https://arxiv.org/abs/2411.03042)
• Keywords: language model
• Abstract: Residual networks, as discrete approximations of Ordinary Differential Equations (ODEs), have inspired significant advancements in neural network design, including multistep methods, high-order methods, and multi-particle dynamical systems. The precision of the solution to ODEs significantly affects parameter optimization, thereby impacting model performance. In this work, we present a series of advanced explorations of Transformer architecture design to minimize the error compared to the true ``solution.'' First, we introduce a predictor-corrector learning framework to minimize truncation errors, which consists of a high-order predictor and a multistep corrector. Second, we propose an exponential moving average-based coefficient learning method to strengthen our higher-order predictor. Extensive experiments on large-scale machine translation, abstractive summarization, language modeling, and natural language understanding benchmarks demonstrate the superiority of our approach. On the WMT'14 English-German and English-French tasks, our model achieved BLEU scores of 30.95 and 44.27, respectively. Furthermore, on the OPUS multilingual machine translation task, our model surpasses a robust 3.8B DeepNet by an average of 2.9 SacreBLEU, using only 1/3 parameters. Notably, it also beats LLama models by 5.7 accuracy points on the LM Harness Evaluation.
• 摘要：残差网络作为常微分方程 (ODE) 的离散近似，激发了神经网络设计的重大进步，包括多步方法、高阶方法和多粒子动力学系统。ODE 解的精度会显著影响参数优化，从而影响模型性能。在这项工作中，我们提出了一系列 Transformer 架构设计的高级探索，以尽量减少与真实“解决方案”相比的误差。首先，我们引入了一个预测器-校正器学习框架来尽量减少截断误差，该框架由一个高阶预测器和一个多步校正器组成。其次，我们提出了一种基于指数移动平均的系数学习方法来加强我们的高阶预测器。在大规模机器翻译、抽象摘要、语言建模和自然语言理解基准上进行的大量实验证明了我们方法的优越性。在 WMT'14 英语-德语和英语-法语任务中，我们的模型分别获得了 30.95 和 44.27 的 BLEU 分数。此外，在 OPUS 多语言机器翻译任务中，我们的模型仅使用 1/3 的参数，就比强大的 3.8B DeepNet 平均高出 2.9 SacreBLEU。值得注意的是，它在 LM Harness 评估中也比 LLama 模型高出 5.7 个准确度点。

Title: VERITAS: A Unified Approach to Reliability Evaluation

• Authors: Rajkumar Ramamurthy, Meghana Arakkal Rajeev, Oliver Molenschot, James Zou, Nazneen Rajani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03300
• Pdf URL: https://arxiv.org/pdf/2411.03300
• Copy Paste: [[2411.03300]] VERITAS: A Unified Approach to Reliability Evaluation(https://arxiv.org/abs/2411.03300)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Large language models (LLMs) often fail to synthesize information from their context to generate an accurate response. This renders them unreliable in knowledge intensive settings where reliability of the output is key. A critical component for reliable LLMs is the integration of a robust fact-checking system that can detect hallucinations across various formats. While several open-access fact-checking models are available, their functionality is often limited to specific tasks, such as grounded question-answering or entailment verification, and they perform less effectively in conversational settings. On the other hand, closed-access models like GPT-4 and Claude offer greater flexibility across different contexts, including grounded dialogue verification, but are hindered by high costs and latency. In this work, we introduce VERITAS, a family of hallucination detection models designed to operate flexibly across diverse contexts while minimizing latency and costs. VERITAS achieves state-of-the-art results considering average performance on all major hallucination detection benchmarks, with $10\%$ increase in average performance when compared to similar-sized models and get close to the performance of GPT4 turbo with LLM-as-a-judge setting.
• 摘要：大型语言模型 (LLM) 通常无法从其上下文中综合信息以生成准确的响应。这使得它们在知识密集型环境中不可靠，因为输出的可靠性是关键。可靠的 LLM 的一个关键组成部分是集成一个强大的事实核查系统，该系统可以检测各种格式的幻觉。虽然有几种开放访问的事实核查模型可用，但它们的功能通常仅限于特定任务，例如有根据的问答或蕴涵验证，并且在对话环境中的表现较差。另一方面，像 GPT-4 和 Claude 这样的封闭访问模型在不同环境中提供了更大的灵活性，包括有根据的对话验证，但受到高成本和延迟的阻碍。在这项工作中，我们介绍了 VERITAS，这是一组幻觉检测模型，旨在灵活地跨不同环境运行，同时最大限度地降低延迟和成本。 VERITAS 在所有主要幻觉检测基准上都取得了最先进的结果，考虑到平均性能，与类似大小的模型相比，平均性能提高了 $10\%$，并接近使用 LLM-as-a-judge 设置的 GPT4 turbo 的性能。

Title: LLMs for Domain Generation Algorithm Detection

• Authors: Reynier Leyva La O, Carlos A. Catania, Tatiana Parlanti
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.03307
• Pdf URL: https://arxiv.org/pdf/2411.03307
• Copy Paste: [[2411.03307]] LLMs for Domain Generation Algorithm Detection(https://arxiv.org/abs/2411.03307)
• Keywords: language model, llm
• Abstract: This work analyzes the use of large language models (LLMs) for detecting domain generation algorithms (DGAs). We perform a detailed evaluation of two important techniques: In-Context Learning (ICL) and Supervised Fine-Tuning (SFT), showing how they can improve detection. SFT increases performance by using domain-specific data, whereas ICL helps the detection model to quickly adapt to new threats without requiring much retraining. We use Meta's Llama3 8B model, on a custom dataset with 68 malware families and normal domains, covering several hard-to-detect schemes, including recent word-based DGAs. Results proved that LLM-based methods can achieve competitive results in DGA detection. In particular, the SFT-based LLM DGA detector outperforms state-of-the-art models using attention layers, achieving 94% accuracy with a 4% false positive rate (FPR) and excelling at detecting word-based DGA domains.
• 摘要：本研究分析了大型语言模型 (LLM) 在检测域生成算法 (DGA) 中的应用。我们对两种重要技术进行了详细评估：上下文学习 (ICL) 和监督微调 (SFT)，展示了它们如何改进检测。SFT 通过使用特定于域的数据来提高性能，而 ICL 则帮助检测模型快速适应新威胁而无需进行大量再训练。我们使用 Meta 的 Llama3 8B 模型，在一个包含 68 个恶意软件家族和正常域的自定义数据集上，涵盖了几种难以检测的方案，包括最近的基于单词的 DGA。结果证明，基于 LLM 的方法可以在 DGA 检测中取得有竞争力的结果。特别是，基于 SFT 的 LLM DGA 检测器优于使用注意层的最先进的模型，准确率为 94%，误报率为 4%，并且在检测基于单词的 DGA 域方面表现出色。