2025-07-17

Title: Subjective Evaluation Profile Analysis of Science Fiction Short Stories and its Critical-Theoretical Significance

Authors: Kazuyoshi Otsuka
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.11582
Pdf URL: https://arxiv.org/pdf/2507.11582
Copy Paste: [[2507.11582]] Subjective Evaluation Profile Analysis of Science Fiction Short Stories and its Critical-Theoretical Significance(https://arxiv.org/abs/2507.11582)
Keywords: language model, llm
Abstract: This study positions large language models (LLMs) as "subjective literary critics" to explore aesthetic preferences and evaluation patterns in literary assessment. Ten Japanese science fiction short stories were translated into English and evaluated by six state-of-the-art LLMs across seven independent sessions. Principal component analysis and clustering techniques revealed significant variations in evaluation consistency ({\alpha} ranging from 1.00 to 0.35) and five distinct evaluation patterns. Additionally, evaluation variance across stories differed by up to 4.5-fold, with TF-IDF analysis confirming distinctive evaluation vocabularies for each model. Our seven-session within-day protocol using an original Science Fiction corpus strategically minimizes external biases, allowing us to observe implicit value systems shaped by RLHF and their influence on literary judgment. These findings suggest that LLMs may possess individual evaluation characteristics similar to human critical schools, rather than functioning as neutral benchmarkers.
摘要：这项研究将大型语言模型（LLM）定位为“主观文学批评家”，以探索文学评估中的美学偏好和评估模式。十个日本科幻短篇小说被翻译成英文，并在七个独立会议上由六个最先进的LLM进行了评估。主成分分析和聚类技术揭示了评估一致性的显着差异（{\ alpha}范围从1.00到0.35）和五种不同的评估模式。此外，跨故事的评估差异差异高达4.5倍，TF-IDF分析确认了每个模型的独特评估词汇。我们使用原始科幻小说的日常协议在日常协议中策略性地使外部偏见最小化，从而使我们能够观察RLHF塑造的隐式价值系统及其对文学判断的影响。这些发现表明，LLM可能具有类似于人类关键学校的个人评估特征，而不是充当中性基准测试器。

Title: MapIQ: Benchmarking Multimodal Large Language Models for Map Question Answering

Authors: Varun Srivastava, Fan Lei, Srija Mukhopadhyay, Vivek Gupta, Ross Maciejewski
Subjects: cs.CL, cs.AI, cs.CV, cs.LG
Abstract URL: https://arxiv.org/abs/2507.11625
Pdf URL: https://arxiv.org/pdf/2507.11625
Copy Paste: [[2507.11625]] MapIQ: Benchmarking Multimodal Large Language Models for Map Question Answering(https://arxiv.org/abs/2507.11625)
Keywords: language model, llm
Abstract: Recent advancements in multimodal large language models (MLLMs) have driven researchers to explore how well these models read data visualizations, e.g., bar charts, scatter plots. More recently, attention has shifted to visual question answering with maps (Map-VQA). However, Map-VQA research has primarily focused on choropleth maps, which cover only a limited range of thematic categories and visual analytical tasks. To address these gaps, we introduce MapIQ, a benchmark dataset comprising 14,706 question-answer pairs across three map types: choropleth maps, cartograms, and proportional symbol maps spanning topics from six distinct themes (e.g., housing, crime). We evaluate multiple MLLMs using six visual analytical tasks, comparing their performance against one another and a human baseline. An additional experiment examining the impact of map design changes (e.g., altered color schemes, modified legend designs, and removal of map elements) provides insights into the robustness and sensitivity of MLLMs, their reliance on internal geographic knowledge, and potential avenues for improving Map-VQA performance.
摘要：多模式大语言模型（MLLM）的最新进展使研究人员探索了这些模型读取数据可视化的程度，例如条形图，散点图。最近，注意力转移到了用地图（MAP-VQA）回答的视觉问题上。但是，MAP-VQA研究主要集中在唱片地图上，该地图仅涵盖有限的主题类别和视觉分析任务。为了解决这些差距，我们介绍了Mapiq，这是一个基准数据集，其中包含三种地图类型的14,706个问题 - 答案对：Choropleth地图，摄影图和比例符号图，该符号涵盖了来自六个不同主题的主题（例如，住房，犯罪）。我们使用六项视觉分析任务评估了多个MLLM，并将其彼此的表现与人类基线进行比较。另一个研究地图设计变化的影响（例如，变化的配色方案，修改的传奇设计和地图元素的去除）提供了对MLLM的鲁棒性和敏感性的见解，它们对内部地理知识的依赖以及潜在的地理效果，以提高MAP-VQA性能。

Title: ExpliCIT-QA: Explainable Code-Based Image Table Question Answering

Authors: Maximiliano Hormazábal Lagos, Álvaro Bueno Sáez, Pedro Alonso Doval, Jorge Alcalde Vesteiro, Héctor Cerezo-Costas
Subjects: cs.CL, cs.AI
Abstract URL: https://arxiv.org/abs/2507.11694
Pdf URL: https://arxiv.org/pdf/2507.11694
Copy Paste: [[2507.11694]] ExpliCIT-QA: Explainable Code-Based Image Table Question Answering(https://arxiv.org/abs/2507.11694)
Keywords: chain-of-thought
Abstract: We present ExpliCIT-QA, a system that extends our previous MRT approach for tabular question answering into a multimodal pipeline capable of handling complex table images and providing explainable answers. ExpliCIT-QA follows a modular design, consisting of: (1) Multimodal Table Understanding, which uses a Chain-of-Thought approach to extract and transform content from table images; (2) Language-based Reasoning, where a step-by-step explanation in natural language is generated to solve the problem; (3) Automatic Code Generation, where Python/Pandas scripts are created based on the reasoning steps, with feedback for handling errors; (4) Code Execution to compute the final answer; and (5) Natural Language Explanation that describes how the answer was computed. The system is built for transparency and auditability: all intermediate outputs, parsed tables, reasoning steps, generated code, and final answers are available for inspection. This strategy works towards closing the explainability gap in end-to-end TableVQA systems. We evaluated ExpliCIT-QA on the TableVQA-Bench benchmark, comparing it with existing baselines. We demonstrated improvements in interpretability and transparency, which open the door for applications in sensitive domains like finance and healthcare where auditing results are critical.
摘要：我们提出了explicit-qa，该系统将我们以前的捷运方法扩展到表达式问题，以回答答案，以便能够处理复杂的表图像并提供可解释的答案。显式-QA遵循一个模块化设计，由以下模块化设计组成：（1）多模式表理解，该理解使用了一项经过经过经过经过经过思考的方法来从表图像中提取和转换内容；（2）基于语言的推理，生成自然语言的分步解释来解决问题；（3）自动代码生成，其中python/pandas脚本是根据推理步骤创建的，并具有处理错误的反馈；（4）执行代码以计算最终答案；（5）描述了如何计算答案的自然语言解释。该系统是为了透明性和可审核性而构建的：所有中间输出，解析表，推理步骤，生成的代码以及最终答案都可以进行检查。该策略可用于缩小端到端TableVQA系统中的解释性差距。我们在TableVQA基准测试基准上评估了explicit-qa，并将其与现有基线进行了比较。我们证明了可解释性和透明度的改善，这为在敏感领域（例如审计结果至关重要的财务和医疗保健）中的应用打开了大门。

Title: CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks

Authors: Meng Li, Timothy M. McPhillips, Dingmin Wang, Shin-Rong Tsai, Bertram Ludäscher
Subjects: cs.CL, cs.AI
Abstract URL: https://arxiv.org/abs/2507.11742
Pdf URL: https://arxiv.org/pdf/2507.11742
Copy Paste: [[2507.11742]] CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks(https://arxiv.org/abs/2507.11742)
Keywords: language model, llm, hallucination
Abstract: Recognizing the information flows and operations comprising data science and machine learning Python notebooks is critical for evaluating, reusing, and adapting notebooks for new tasks. Investigating a notebook via re-execution often is impractical due to the challenges of resolving data and software dependencies. While Large Language Models (LLMs) pre-trained on large codebases have demonstrated effectiveness in understanding code without running it, we observe that they fail to understand some realistic notebooks due to hallucinations and long-context challenges. To address these issues, we propose a notebook understanding task yielding an information flow graph and corresponding cell execution dependency graph for a notebook, and demonstrate the effectiveness of a pincer strategy that uses limited syntactic analysis to assist full comprehension of the notebook using an LLM. Our Capture and Resolve Assisted Bounding Strategy (CRABS) employs shallow syntactic parsing and analysis of the abstract syntax tree (AST) to capture the correct interpretation of a notebook between lower and upper estimates of the inter-cell I/O sets, then uses an LLM to resolve remaining ambiguities via cell-by-cell zero-shot learning, thereby identifying the true data inputs and outputs of each cell. We evaluate and demonstrate the effectiveness of our approach using an annotated dataset of 50 representative, highly up-voted Kaggle notebooks that together represent 3454 actual cell inputs and outputs. The LLM correctly resolves 1397 of 1425 (98%) ambiguities left by analyzing the syntactic structure of these notebooks. Across 50 notebooks, CRABS achieves average F1 scores of 98% identifying cell-to-cell information flows and 99% identifying transitive cell execution dependencies.
摘要：识别包括数据科学和机器学习Python笔记本的信息流和操作对于评估，重复使用和调整笔记本的新任务至关重要。由于解决数据和软件依赖性的挑战，通过重新执行调查笔记本通常是不切实际的。尽管在大型代码库中预先培训的大型语言模型（LLM）在不运行代码的情况下表现出了有效性，但我们观察到，由于幻觉和长期文本挑战，他们无法理解一些现实的笔记本。为了解决这些问题，我们提出了一个笔记本理解任务，该任务产生了信息流程图和相应的单元执行依赖图，并证明了使用有限的句法分析来帮助使用llm的笔记本充分理解笔记本。 Our Capture and Resolve Assisted Bounding Strategy (CRABS) employs shallow syntactic parsing and analysis of the abstract syntax tree (AST) to capture the correct interpretation of a notebook between lower and upper estimates of the inter-cell I/O sets, then uses an LLM to resolve remaining ambiguities via cell-by-cell zero-shot learning, thereby identifying the true data inputs and outputs of each cell.我们使用50个代表性，高度投票的Kaggle笔记本的注释数据集评估并证明我们的方法的有效性，共同代表3454个实际的单元格输入和输出。通过分析这些笔记本的句法结构，LLM正确解决了1425（98％）的1397（98％）。在50个笔记本中，螃蟹达到了98％的平均F1分数，识别细胞对细胞信息流量，而99％的F1得分可以识别透视细胞执行依赖性。

Title: Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models

Authors: Dante Campregher, Yanxu Chen, Sander Hoffman, Maria Heuss
Subjects: cs.CL, cs.AI, cs.LG
Abstract URL: https://arxiv.org/abs/2507.11809
Pdf URL: https://arxiv.org/pdf/2507.11809
Copy Paste: [[2507.11809]] Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models(https://arxiv.org/abs/2507.11809)
Keywords: language model, llm
Abstract: This paper presents a reproducibility study examining how Large Language Models (LLMs) manage competing factual and counterfactual information, focusing on the role of attention heads in this process. We attempt to reproduce and reconcile findings from three recent studies by Ortu et al., Yu, Merullo, and Pavlick and McDougall et al. that investigate the competition between model-learned facts and contradictory context information through Mechanistic Interpretability tools. Our study specifically examines the relationship between attention head strength and factual output ratios, evaluates competing hypotheses about attention heads' suppression mechanisms, and investigates the domain specificity of these attention patterns. Our findings suggest that attention heads promoting factual output do so via general copy suppression rather than selective counterfactual suppression, as strengthening them can also inhibit correct facts. Additionally, we show that attention head behavior is domain-dependent, with larger models exhibiting more specialized and category-sensitive patterns.
摘要：本文提出了一项可重复性研究，研究了大型语言模型（LLM）如何管理竞争的事实和反事实信息，重点是注意力头在此过程中的作用。我们试图从Ortu等，Yu，Merullo和Pavlick和McDougall等人的三项研究中复制和调和发现。通过机械性解释性工具，研究了模型学习的事实和矛盾的上下文信息之间的竞争。我们的研究专门研究了注意力头强度和事实输出比之间的关系，评估了有关注意力头的抑制机制的竞争假设，并研究了这些注意力模式的领域特异性。我们的发现表明，促进事实输出的注意力通过一般拷贝抑制而不是选择性的反事实抑制来，因为加强它们也可以抑制正确的事实。此外，我们表明注意力头部行为依赖于域，较大的模型表现出更专业和类别敏感的模式。

Title: Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential

Authors: Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, Mehrdad Farajtabar
Subjects: cs.CL, cs.LG
Abstract URL: https://arxiv.org/abs/2507.11851
Pdf URL: https://arxiv.org/pdf/2507.11851
Copy Paste: [[2507.11851]] Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential(https://arxiv.org/abs/2507.11851)
Keywords: language model, llm, chat
Abstract: Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especially during later stages of generation when the direction and semantics of text are relatively certain. In this work, we propose a novel framework that leverages the inherent knowledge of vanilla autoregressive language models about future tokens, combining techniques to realize this potential and enable simultaneous prediction of multiple subsequent tokens. Our approach introduces several key innovations: (1) a masked-input formulation where multiple future tokens are jointly predicted from a common prefix; (2) a gated LoRA formulation that preserves the original LLM's functionality, while equipping it for multi-token prediction; (3) a lightweight, learnable sampler module that generates coherent sequences from the predicted future tokens; (4) a set of auxiliary training losses, including a consistency loss, to enhance the coherence and accuracy of jointly generated tokens; and (5) a speculative generation strategy that expands tokens quadratically in the future while maintaining high fidelity. Our method achieves significant speedups through supervised fine-tuning on pretrained models. For example, it generates code and math nearly 5x faster, and improves general chat and knowledge tasks by almost 2.5x. These gains come without any loss in quality.
摘要：自回归语言模型受其固有的顺序性质的约束，一次产生一个令牌。该范式限制了推理速度和并行性，尤其是在生成阶段的文本方向和语义相对确定时。在这项工作中，我们提出了一个新颖的框架，该框架利用了关于未来令牌的香草自回归语言模型的固有知识，结合了技术来实现这一潜力并能够同时预测随后多个令牌。我们的方法介绍了几项关键创新：（1）蒙版输入公式，其中多个未来代币可以从共同的前缀共同预测；（2）一个封闭式的Lora公式，可保留原始LLM的功能，同时将其装备用于多token预测；（3）一个轻巧的可学习的采样器模块，该模块从预测的未来令牌中生成相干序列；（4）一组辅助训练损失，包括一致性损失，以提高共同产生的令牌的连贯性和准确性；（5）一种投机策略，将来在保持高保真度的同时，在将来四次扩展令牌。我们的方法通过对验证的模型进行的微调来实现重大加速。例如，它生成代码和数学的速度近5倍，并将一般聊天和知识任务提高了几乎2.5倍。这些收益质量不会损失。

Title: LLMs Encode Harmfulness and Refusal Separately

Authors: Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, Weiyan Shi
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.11878
Pdf URL: https://arxiv.org/pdf/2507.11878
Copy Paste: [[2507.11878]] LLMs Encode Harmfulness and Refusal Separately(https://arxiv.org/abs/2507.11878)
Keywords: llm
Abstract: LLMs are trained to refuse harmful instructions, but do they truly understand harmfulness beyond just refusing? Prior work has shown that LLMs' refusal behaviors can be mediated by a one-dimensional subspace, i.e., a refusal direction. In this work, we identify a new dimension to analyze safety mechanisms in LLMs, i.e., harmfulness, which is encoded internally as a separate concept from refusal. There exists a harmfulness direction that is distinct from the refusal direction. As causal evidence, steering along the harmfulness direction can lead LLMs to interpret harmless instructions as harmful, but steering along the refusal direction tends to elicit refusal responses directly without reversing the model's judgment on harmfulness. Furthermore, using our identified harmfulness concept, we find that certain jailbreak methods work by reducing the refusal signals without reversing the model's internal belief of harmfulness. We also find that adversarially finetuning models to accept harmful instructions has minimal impact on the model's internal belief of harmfulness. These insights lead to a practical safety application: The model's latent harmfulness representation can serve as an intrinsic safeguard (Latent Guard) for detecting unsafe inputs and reducing over-refusals that is robust to finetuning attacks. For instance, our Latent Guard achieves performance comparable to or better than Llama Guard 3 8B, a dedicated finetuned safeguard model, across different jailbreak methods. Our findings suggest that LLMs' internal understanding of harmfulness is more robust than their refusal decision to diverse input instructions, offering a new perspective to study AI safety
摘要：LLM经过训练以拒绝有害说明，但是他们不仅拒绝拒绝有害吗？先前的工作表明，LLMS的拒绝行为可以由一维子空间（即拒绝方向）介导。在这项工作中，我们确定了一个新的维度来分析LLMS中的安全机制，即有害性，该机制在内部编码为与拒绝的单独概念。存在一个有害方向，与拒绝方向不同。作为因果证据，沿着有害方向的方向转向可以导致LLM将无害指示解释为有害指示，但是沿着拒绝方向转向倾向于直接引起拒绝反应，而不会反转模型对危害的判断。此外，使用我们确定的有害概念，我们发现某些越狱方法通过减少拒绝信号而不反转模型的内部信念来起作用。我们还发现，接受有害指示的对抗性训练模型对该模型对有害性的内部信念的影响很小。这些见解导致了实际的安全应用：该模型的潜在有害性表示可以用作固有的保障（潜在警卫），用于检测不安全的输入并减少对捕获攻击的强大的过度费用。例如，我们的潜在警卫在不同的越狱方法中获得了与Llama Guard 3 8B相当或更好的表现。我们的发现表明，LLMS对有害性的内部理解比拒绝不同的投入说明更强大，提供了一种新的观点来研究AI安全性

Title: Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language Models

Authors: Bo Zeng, Chenyang Lyu, Sinuo Liu, Mingyan Zeng, Minghao Wu, Xuanfan Ni, Tianqi Shi, Yu Zhao, Yefeng Liu, Chenyu Zhu, Ruizhe Li, Jiahui Geng, Qing Li, Yu Tong, Longyue Wang, Weihua Luo, Kaifu Zhang
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.11882
Pdf URL: https://arxiv.org/pdf/2507.11882
Copy Paste: [[2507.11882]] Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language Models(https://arxiv.org/abs/2507.11882)
Keywords: language model, llm, prompt
Abstract: Instruction-following capability has become a major ability to be evaluated for Large Language Models (LLMs). However, existing datasets, such as IFEval, are either predominantly monolingual and centered on English or simply machine translated to other languages, limiting their applicability in multilingual contexts. In this paper, we present an carefully-curated extension of IFEval to a localized multilingual version named Marco-Bench-MIF, covering 30 languages with varying levels of localization. Our benchmark addresses linguistic constraints (e.g., modifying capitalization requirements for Chinese) and cultural references (e.g., substituting region-specific company names in prompts) via a hybrid pipeline combining translation with verification. Through comprehensive evaluation of 20+ LLMs on our Marco-Bench-MIF, we found that: (1) 25-35% accuracy gap between high/low-resource languages, (2) model scales largely impact performance by 45-60% yet persists script-specific challenges, and (3) machine-translated data underestimates accuracy by7-22% versus localized data. Our analysis identifies challenges in multilingual instruction following, including keyword consistency preservation and compositional constraint adherence across languages. Our Marco-Bench-MIF is available at this https URL.
摘要：指导跟随能力已成为对大语言模型（LLM）评估的主要能力。但是，现有的数据集（例如Ifeval）主要是单语言，并以英语为中心，或者简单地转换为其他语言，从而将其适用性限制在多语言上下文中。在本文中，我们提出了IFEVAL经过精心策划的扩展到名为Marco-Bench-Mif的本地化多语言版本，涵盖了30种具有不同定位水平的语言。我们的基准测试通过混合管道结合翻译与验证，解决了语言限制（例如，在提示中修改资本化要求）和文化参考（例如，提示中的替代区域特定公司名称）。 Through comprehensive evaluation of 20+ LLMs on our Marco-Bench-MIF, we found that: (1) 25-35% accuracy gap between high/low-resource languages, (2) model scales largely impact performance by 45-60% yet persists script-specific challenges, and (3) machine-translated data underestimates accuracy by7-22% versus localized data.我们的分析确定了以下多语言教学的挑战，包括关键字一致性保存和跨语言的组成约束依从性。我们的Marco-Bench-Mif可在此HTTPS URL上找到。

Title: A Survey of Deep Learning for Geometry Problem Solving

Authors: Jianzhe Ma, Wenxuan Wang, Qin Jin
Subjects: cs.CL, cs.AI, cs.CV, cs.LG
Abstract URL: https://arxiv.org/abs/2507.11936
Pdf URL: https://arxiv.org/pdf/2507.11936
Copy Paste: [[2507.11936]] A Survey of Deep Learning for Geometry Problem Solving(https://arxiv.org/abs/2507.11936)
Keywords: language model
Abstract: Geometry problem solving is a key area of mathematical reasoning, which is widely involved in many important fields such as education, mathematical ability assessment of artificial intelligence, and multimodal ability assessment. In recent years, the rapid development of deep learning technology, especially the rise of multimodal large language models, has triggered a widespread research boom. This paper provides a survey of the applications of deep learning in geometry problem solving, including (i) a comprehensive summary of the relevant tasks in geometry problem solving; (ii) a thorough review of related deep learning methods; (iii) a detailed analysis of evaluation metrics and methods; and (iv) a critical discussion of the current challenges and future directions that can be explored. Our goal is to provide a comprehensive and practical reference of deep learning for geometry problem solving to promote further developments in this field. We create a continuously updated list of papers on GitHub: this https URL.
摘要：几何问题解决是数学推理的关键领域，它广泛参与了许多重要领域，例如教育，数学能力评估人工智能和多模式能力评估。近年来，深度学习技术的快速发展，尤其是多模式大语言模型的兴起，引发了广泛的研究繁荣。本文对深度学习在几何问题解决问题中的应用进行了调查，包括（i）几何问题解决方案中相关任务的全面摘要；（ii）对相关深度学习方法的彻底回顾；（iii）评估指标和方法的详细分析；（iv）对当前挑战和未来方向的批判性讨论。我们的目标是为解决几何问题的深度学习提供全面而实用的参考，以促进该领域的进一步发展。我们在Github上创建了一个不断更新的论文列表：此HTTPS URL。

Title: POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering

Authors: Yichen Xu, Liangyu Chen, Liang Zhang, Wenxuan Wang, Qin Jin
Subjects: cs.CL, cs.AI, cs.CV, cs.MM
Abstract URL: https://arxiv.org/abs/2507.11939
Pdf URL: https://arxiv.org/pdf/2507.11939
Copy Paste: [[2507.11939]] POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering(https://arxiv.org/abs/2507.11939)
Keywords: language model, llm
Abstract: Charts are a universally adopted medium for interpreting and communicating data. However, existing chart understanding benchmarks are predominantly English-centric, limiting their accessibility and applicability to global audiences. In this paper, we present PolyChartQA, the first large-scale multilingual chart question answering benchmark covering 22,606 charts and 26,151 question-answering pairs across 10 diverse languages. PolyChartQA is built using a decoupled pipeline that separates chart data from rendering code, allowing multilingual charts to be flexibly generated by simply translating the data and reusing the code. We leverage state-of-the-art LLM-based translation and enforce rigorous quality control in the pipeline to ensure the linguistic and semantic consistency of the generated multilingual charts. PolyChartQA facilitates systematic evaluation of multilingual chart understanding. Experiments on both open- and closed-source large vision-language models reveal a significant performance gap between English and other languages, especially low-resource ones with non-Latin scripts. This benchmark lays a foundation for advancing globally inclusive vision-language models.
摘要：图表是通用解释和通信数据的普遍采用的媒介。但是，现有的图表理解基准主要是以英语为中心，从而限制了其对全球受众的可访问性和适用性。在本文中，我们介绍了Polychartqa，这是第一个大规模的多语言图表问题回答基准，涵盖了10种不同语言的22,606个图表和26,151个提问对。 PolyChartQA是使用将图表数据与渲染代码区分开的解耦管道构建的，从而可以通过简单地翻译数据并重用代码来灵活地生成多语言图表。我们利用最新的基于LLM的翻译并在管道中执行严格的质量控制，以确保生成的多语言图表的语言和语义一致性。 PolyChartQA促进了多语言图表理解的系统评估。开放式和封闭源的大型视觉模型的实验揭示了英语和其他语言之间的显着性能差距，尤其是具有非LATIN脚本的低资源语言。该基准为推进全球包容性视觉语言模型奠定了基础。

Title: BlockBPE: Parallel BPE Tokenization

Authors: Amos You
Subjects: cs.CL, cs.DC
Abstract URL: https://arxiv.org/abs/2507.11941
Pdf URL: https://arxiv.org/pdf/2507.11941
Copy Paste: [[2507.11941]] BlockBPE: Parallel BPE Tokenization(https://arxiv.org/abs/2507.11941)
Keywords: language model
Abstract: Tokenization is a critical preprocessing step in large language model pipelines, yet widely-used implementations remain CPU-bound and suboptimal for batch inference workflows on GPU. We present BlockBPE, a parallel GPU implementation of byte-pair encoding (BPE) that achieves near linear-time complexity under realistic assumptions and is optimized for high-throughput, batch inference. Unlike existing Rust-based tokenizers such as HuggingFace Tokenizers or OpenAI's tiktoken-whose runtimes are dominated by Regex pre-tokenization and exhibit $O(n \log n)$ runtime-BlockBPE eliminates the Regex pre-tokenization which leads to small loss in generation quality, but enables highly parallelized token merges within thread blocks, reducing overall complexity to $O(nd)$ where $d \ll n$. On high-batch inference workloads, BlockBPE achieves up to 2x higher throughput than tiktoken and 2.5x over HuggingFace Tokenizers.
摘要：令牌化是大语言模型管道中的关键预处理步骤，但是广泛使用的实现仍然是CPU结合的，对于GPU上的批处理推理工作流程而言。我们提出了BlockBPE，这是字节对编码（BPE）的平行GPU实现，该实现在现实假设下实现了接近线性时间复杂性，并针对高通量，批处理推断进行了优化。 Unlike existing Rust-based tokenizers such as HuggingFace Tokenizers or OpenAI's tiktoken-whose runtimes are dominated by Regex pre-tokenization and exhibit $O(n \log n)$ runtime-BlockBPE eliminates the Regex pre-tokenization which leads to small loss in generation quality, but enables highly parallelized token merges within thread blocks, reducing overall complexity to $O(nd)$ where $ d \ ll n $。在高批量推理工作负载上，BlockBPE的吞吐量比Tiktoken高达2倍，而HuggingFace Tokenizers的吞吐量高2.5倍。

Title: DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression

Authors: Yi Zhao, Zuchao Li, Hai Zhao, Baoyuan Qi, Guoming Liu
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.11942
Pdf URL: https://arxiv.org/pdf/2507.11942
Copy Paste: [[2507.11942]] DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression(https://arxiv.org/abs/2507.11942)
Keywords: llm, prompt
Abstract: Task-agnostic prompt compression leverages the redundancy in natural language to reduce computational overhead and enhance information density within prompts, especially in long-context scenarios. Existing methods predominantly rely on information entropy as the metric to compress lexical units, aiming to achieve minimal information loss. However, these approaches overlook two critical aspects: (i) the importance of attention-critical tokens at the algorithmic level, and (ii) shifts in information entropy during the compression process. Motivated by these challenges, we propose a dynamic attention-aware approach for task-agnostic prompt compression (DAC). This approach effectively integrates entropy and attention information, dynamically sensing entropy shifts during compression to achieve fine-grained prompt compression. Extensive experiments across various domains, including LongBench, GSM8K, and BBH, show that DAC consistently yields robust and substantial improvements across a diverse range of tasks and LLMs, offering compelling evidence of its efficacy.
摘要：任务不合时宜的提示压缩将利用自然语言的冗余，以减少计算开销并增强提示内的信息密度，尤其是在长篇小说方案中。现有方法主要依赖于信息熵作为压缩词汇单元的指标，旨在实现最小的信息损失。但是，这些方法忽略了两个关键方面：（i）算法级别的关注关键令牌的重要性，以及（ii）在压缩过程中信息熵的变化。在这些挑战的推动下，我们提出了一种动态的注意力感知方法，以进行任务不合时宜的及时压缩（DAC）。这种方法有效地整合了熵和注意力信息，在压缩过程中动态感应熵变化，以实现细粒度的及时压缩。包括Longbench，GSM8K和BBH在内的各个领域进行的广泛实验表明，DAC始终在各种任务和LLMS范围内实现强大而实质性的改进，从而提供了令人信服的证据。

Title: IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

Authors: Yi Zhao, Zuchao Li, Hai Zhao
Subjects: cs.CL, cs.LG
Abstract URL: https://arxiv.org/abs/2507.11953
Pdf URL: https://arxiv.org/pdf/2507.11953
Copy Paste: [[2507.11953]] IAM: Efficient Inference through Attention Mapping between Different-scale LLMs(https://arxiv.org/abs/2507.11953)
Keywords: llm, long context
Abstract: LLMs encounter significant challenges in resource consumption nowadays, especially with long contexts. Despite extensive efforts dedicate to enhancing inference efficiency, these methods primarily exploit internal sparsity within the models, without leveraging external information for optimization. We identify the high similarity of attention matrices across different-scale LLMs, which offers a novel perspective for optimization. We first conduct a comprehensive analysis of how to measure similarity, how to select mapping Layers and whether mapping is consistency. Based on these insights, we introduce the IAM framework, which achieves dual benefits of accelerated attention computation and reduced KV cache usage by performing attention mapping between small and large LLMs. Our experimental results demonstrate that IAM can accelerate prefill by 15% and reduce KV cache usage by 22.1% without appreciably sacrificing performance. Experiments on different series of models show the generalizability of IAM. Importantly, it is also orthogonal to many existing KV cache optimization methods, making it a versatile addition to the current toolkit for enhancing LLM efficiency.
摘要：如今，LLMS在资源消耗中遇到了重大挑战，尤其是在漫长的背景下。尽管大力致力于提高推论效率，但这些方法主要利用模型中的内部稀疏性，而无需利用外部信息进行优化。我们确定了不同尺度LLM的注意力矩阵的高度相似性，这为优化提供了一种新颖的观点。我们首先对如何测量相似性，如何选择映射层以及映射是否是一致性进行了全面分析。基于这些见解，我们介绍了IAM框架，该框架通过在小型和大型LLM之间进行注意力映射，从而实现了加速注意计算的双重好处，并减少了KV缓存的使用。我们的实验结果表明，IAM可以加速预填充15％，并在不牺牲性能的情况下将KV缓存使用量减少22.1％。不同系列模型的实验显示了IAM的普遍性。重要的是，它也与许多现有的KV缓存优化方法正交，这使其成为当前工具包的多功能，以提高LLM效率。

Title: The benefits of query-based KGQA systems for complex and temporal questions in LLM era

Authors: Artem Alekseev, Mikhail Chaichuk, Miron Butko, Alexander Panchenko, Elena Tutubalina, Oleg Somov
Subjects: cs.CL, cs.LG
Abstract URL: https://arxiv.org/abs/2507.11954
Pdf URL: https://arxiv.org/pdf/2507.11954
Copy Paste: [[2507.11954]] The benefits of query-based KGQA systems for complex and temporal questions in LLM era(https://arxiv.org/abs/2507.11954)
Keywords: language model, llm
Abstract: Large language models excel in question-answering (QA) yet still struggle with multi-hop reasoning and temporal questions. Query-based knowledge graph QA (KGQA) offers a modular alternative by generating executable queries instead of direct answers. We explore multi-stage query-based framework for WikiData QA, proposing multi-stage approach that enhances performance on challenging multi-hop and temporal benchmarks. Through generalization and rejection studies, we evaluate robustness across multi-hop and temporal QA datasets. Additionally, we introduce a novel entity linking and predicate matching method using CoT reasoning. Our results demonstrate the potential of query-based multi-stage KGQA framework for improving multi-hop and temporal QA with small language models. Code and data: this https URL
摘要：大型语言模型在提问（QA）中表现出色，但仍在多跳上的推理和时间问题上挣扎。基于查询的知识图QA（KGQA）通过生成可执行查询而不是直接答案提供了模块化替代方案。我们探索了Wikidata QA的基于多阶段查询的框架，提出了多阶段方法，以提高挑战性多跳和时间基准的性能。通过概括和排斥研究，我们评估了多跳和时间质量检查数据集的鲁棒性。此外，我们使用COT推理介绍了一种新颖的实体链接和谓词匹配方法。我们的结果证明了基于查询的多阶段KGQA框架的潜力，可以通过小语言模型改善多跳和QA。代码和数据：此HTTPS URL

Title: PoTPTQ: A Two-step Power-of-Two Post-training for LLMs

Authors: Xinyu Wang, Vahid Partovi Nia, Peng Lu, Jerry Huang, Xiao-Wen Chang, Boxing Chen, Yufei Cui
Subjects: cs.CL, cs.AI, cs.LG
Abstract URL: https://arxiv.org/abs/2507.11959
Pdf URL: https://arxiv.org/pdf/2507.11959
Copy Paste: [[2507.11959]] PoTPTQ: A Two-step Power-of-Two Post-training for LLMs(https://arxiv.org/abs/2507.11959)
Keywords: language model, llm
Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing (NLP) tasks. However, their deployment is challenging due to the substantial computational resources required. Power-of-two (PoT) quantization is a general tool to counteract this difficulty. Albeit previous works on PoT quantization can be efficiently dequantized on CPUs using fixed-point addition, it showed less effectiveness on GPUs. The reason is entanglement of the sign bit and sequential bit manipulations needed for dequantization. We propose a novel POT quantization framework for LLM weights that (i) outperforms state-of-the-art accuracy in extremely low-precision number formats, and (ii) enables faster inference through more efficient dequantization. To maintain the accuracy of the quantized model, we introduce a two-step post-training algorithm: (i) initialize the quantization scales with a robust starting point, and (ii) refine these scales using a minimal calibration set. The performance of our PoT post-training algorithm surpasses the current state-of-the-art in integer quantization, particularly at low precisions such as 2- and 3-bit formats. Our PoT quantization accelerates the dequantization step required for the floating point inference and leads to $3.67\times$ speed up on a NVIDIA V100, and $1.63\times$ on a NVIDIA RTX 4090, compared to uniform integer dequantization.
摘要：大型语言模型（LLMS）在各种自然语言处理（NLP）任务中表现出色。但是，由于所需的大量计算资源，它们的部署具有挑战性。两个量子（锅）量化是抵消这一难度的一般工具。尽管可以使用固定点的添加在CPU上有效地在CPU上有效地取消了有关POT量化的工作，但对GPU的有效性较小。原因是符号位的纠缠和取消化所需的顺序位操作。我们为LLM权重的新型锅量化框架（i）（i）以极低精确的数字格式优于最先进的精度，（ii）通过更有效的去量化来更快地推断。为了维持量化模型的准确性，我们引入了两步训练后算法：（i）使用稳健的起点初始化量化标尺，（ii）使用最小的校准集来完善这些量表。我们的锅后训练算法的性能超过了整数量化的当前最新技术，尤其是在低精度（例如2位和3位格式）下。我们的锅量化加速了浮点推理所需的去量化步骤，并在NVIDIA V100上提高了$ 3.67 \ times $加快速度，而NVIDIA RTX 4090上的$ 1.63 \ times $，与统一的整数降级相比。

Title: Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation

Authors: Ziyu Ge, Gabriel Chua, Leanne Tan, Roy Ka-Wei Lee
Subjects: cs.CL, cs.AI, cs.CY
Abstract URL: https://arxiv.org/abs/2507.11966
Pdf URL: https://arxiv.org/pdf/2507.11966
Copy Paste: [[2507.11966]] Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation(https://arxiv.org/abs/2507.11966)
Keywords: language model, llm, prompt
Abstract: As online communication increasingly incorporates under-represented languages and colloquial dialects, standard translation systems often fail to preserve local slang, code-mixing, and culturally embedded markers of harmful speech. Translating toxic content between low-resource language pairs poses additional challenges due to scarce parallel data and safety filters that sanitize offensive expressions. In this work, we propose a reproducible, two-stage framework for toxicity-preserving translation, demonstrated on a code-mixed Singlish safety corpus. First, we perform human-verified few-shot prompt engineering: we iteratively curate and rank annotator-selected Singlish-target examples to capture nuanced slang, tone, and toxicity. Second, we optimize model-prompt pairs by benchmarking several large language models using semantic similarity via direct and back-translation. Quantitative human evaluation confirms the effectiveness and efficiency of our pipeline. Beyond improving translation quality, our framework contributes to the safety of multicultural LLMs by supporting culturally sensitive moderation and benchmarking in low-resource contexts. By positioning Singlish as a testbed for inclusive NLP, we underscore the importance of preserving sociolinguistic nuance in real-world applications such as content moderation and regional platform governance.
摘要：随着在线交流越来越多地融合了代表性不足的语言和口语方言，标准翻译系统通常无法保留本地的语，混合代码和有害语音的文化嵌入标记。在低资源语言对之间翻译有毒内容，由于稀缺的平行数据和安全过滤器，会带来其他挑战，这些挑战可以消毒进攻表达式。在这项工作中，我们提出了一个可再现的，两阶段的毒性含量翻译框架，该框架在代码混合的Singlish Safety语料库中证明了这一点。首先，我们执行人类验证的少量及时工程：我们迭代地策划和等级注释者选择的Singlish-Target示例，以捕获细微的语，音调和毒性。其次，我们通过直接和反向翻译使用语义相似性对几个大语言模型进行基准测试，从而优化了模型 - 推出对。定量人类评估证实了我们管道的有效性和效率。除了提高翻译质量外，我们的框架还通过在低资源背景下支持文化敏感的适度和基准测试来有助于多元文化LLM的安全。通过将Singlish定位为包容性NLP的测试床，我们强调了在现实世界中（例如内容审核和区域平台治理）中保留社会语言细微差别的重要性。

Title: Graph Representations for Reading Comprehension Analysis using Large Language Model and Eye-Tracking Biomarker

Authors: Yuhong Zhang, Jialu Li, Shilai Yang, Yuchen Xu, Gert Cauwenberghs, Tzyy-Ping Jung
Subjects: cs.CL, q-bio.NC
Abstract URL: https://arxiv.org/abs/2507.11972
Pdf URL: https://arxiv.org/pdf/2507.11972
Copy Paste: [[2507.11972]] Graph Representations for Reading Comprehension Analysis using Large Language Model and Eye-Tracking Biomarker(https://arxiv.org/abs/2507.11972)
Keywords: language model, llm, prompt, agent
Abstract: Reading comprehension is a fundamental skill in human cognitive development. With the advancement of Large Language Models (LLMs), there is a growing need to compare how humans and LLMs understand language across different contexts and apply this understanding to functional tasks such as inference, emotion interpretation, and information retrieval. Our previous work used LLMs and human biomarkers to study the reading comprehension process. The results showed that the biomarkers corresponding to words with high and low relevance to the inference target, as labeled by the LLMs, exhibited distinct patterns, particularly when validated using eye-tracking data. However, focusing solely on individual words limited the depth of understanding, which made the conclusions somewhat simplistic despite their potential significance. This study used an LLM-based AI agent to group words from a reading passage into nodes and edges, forming a graph-based text representation based on semantic meaning and question-oriented prompts. We then compare the distribution of eye fixations on important nodes and edges. Our findings indicate that LLMs exhibit high consistency in language understanding at the level of graph topological structure. These results build on our previous findings and offer insights into effective human-AI co-learning strategies.
摘要：阅读理解是人类认知发展的基本技能。随着大语言模型（LLM）的发展，越来越需要比较人类和LLM在不同环境中如何理解语言，并将这种理解应用于推理，情感解释和信息检索等功能任务。我们以前的工作使用LLM和人类生物标志物来研究阅读理解过程。结果表明，与LLMS标记的与推理目标高和低相关的单词相对应的生物标志物表现出不同的模式，尤其是在使用眼睛跟踪数据验证的情况下。但是，仅专注于单个单词限制了理解的深度，这使得结论具有潜在的意义，但结论有些简单。这项研究使用基于LLM的AI代理将阅读段的单词分组为节点和边缘，从而形成了基于语义含义和面向问题的提示的基于图的文本表示。然后，我们比较重要节点和边缘上的眼睛固定分布。我们的发现表明，LLM在图形拓扑结构水平上的语言理解表现出很高的一致性。这些结果以我们以前的发现为基础，并提供了对有效的人类共同学习策略的见解。

Title: Value-Based Large Language Model Agent Simulation for Mutual Evaluation of Trust and Interpersonal Closeness

Authors: Yuki Sakamoto, Takahisa Uchida, Hiroshi Ishiguro
Subjects: cs.CL, cs.MA
Abstract URL: https://arxiv.org/abs/2507.11979
Pdf URL: https://arxiv.org/pdf/2507.11979
Copy Paste: [[2507.11979]] Value-Based Large Language Model Agent Simulation for Mutual Evaluation of Trust and Interpersonal Closeness(https://arxiv.org/abs/2507.11979)
Keywords: language model, llm, prompt, agent
Abstract: Large language models (LLMs) have emerged as powerful tools for simulating complex social phenomena using human-like agents with specific traits. In human societies, value similarity is important for building trust and close relationships; however, it remains unexplored whether this principle holds true in artificial societies comprising LLM agents. Therefore, this study investigates the influence of value similarity on relationship-building among LLM agents through two experiments. First, in a preliminary experiment, we evaluated the controllability of values in LLMs to identify the most effective model and prompt design for controlling the values. Subsequently, in the main experiment, we generated pairs of LLM agents imbued with specific values and analyzed their mutual evaluations of trust and interpersonal closeness following a dialogue. The experiments were conducted in English and Japanese to investigate language dependence. The results confirmed that pairs of agents with higher value similarity exhibited greater mutual trust and interpersonal closeness. Our findings demonstrate that the LLM agent simulation serves as a valid testbed for social science theories, contributes to elucidating the mechanisms by which values influence relationship building, and provides a foundation for inspiring new theories and insights into the social sciences.
摘要：大型语言模型（LLM）已成为使用具有特定特征的类似人类的药物来模拟复杂的社会现象的强大工具。在人类社会中，价值相似性对于建立信任和亲密关系很重要。但是，该原理在包括LLM代理的人工社会中是否正确，仍然没有探索。因此，本研究通过两个实验研究了价值相似性对LLM代理之间关系建设的影响。首先，在初步实验中，我们评估了LLM中值的可控性，以确定最有效的模型和及时设计以控制值。随后，在主要实验中，我们生成了一对具有特定价值的LLM代理，并在对话后分析了他们对信任和人际亲密关系的相互评估。实验是用英语和日语进行的，以调查语言依赖性。结果证实，具有较高价值相似性的代理对表现出更大的相互信任和人际亲密关系。我们的发现表明，LLM代理模拟是社会科学理论的有效测试，有助于阐明价值观影响关系建立的机制，并为激发新理论和洞察力的社会科学提供了基础。

Title: Simplifications are Absolutists: How Simplified Language Reduces Word Sense Awareness in LLM-Generated Definitions

Authors: Lukas Ellinger, Miriam Anschütz, Georg Groh
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.11981
Pdf URL: https://arxiv.org/pdf/2507.11981
Copy Paste: [[2507.11981]] Simplifications are Absolutists: How Simplified Language Reduces Word Sense Awareness in LLM-Generated Definitions(https://arxiv.org/abs/2507.11981)
Keywords: language model, gpt, llm, prompt
Abstract: Large Language Models (LLMs) can provide accurate word definitions and explanations for any context. However, the scope of the definition changes for different target groups, like children or language learners. This is especially relevant for homonyms, words with multiple meanings, where oversimplification might risk information loss by omitting key senses, potentially misleading users who trust LLM outputs. We investigate how simplification impacts homonym definition quality across three target groups: Normal, Simple, and ELI5. Using two novel evaluation datasets spanning multiple languages, we test DeepSeek v3, Llama 4 Maverick, Qwen3-30B A3B, GPT-4o mini, and Llama 3.1 8B via LLM-as-Judge and human annotations. Our results show that simplification drastically degrades definition completeness by neglecting polysemy, increasing the risk of misunderstanding. Fine-tuning Llama 3.1 8B with Direct Preference Optimization substantially improves homonym response quality across all prompt types. These findings highlight the need to balance simplicity and completeness in educational NLP to ensure reliable, context-aware definitions for all learners.
摘要：大型语言模型（LLMS）可以为任何上下文提供准确的单词定义和解释。但是，定义的范围改变了不同目标群体，例如儿童或语言学习者。这与同音词，具有多种含义的单词尤其重要，在这些单词中，过度简化可能会通过省略关键感官，可能误导LLM输出的用户来冒险丢失。我们研究了简化如何影响三个目标组的同义定义质量：正常，简单和ELI5。使用两个跨越多种语言的新型评估数据集，我们通过LLM-AS-AS-Gudge和人类注释来测试DeepSeek V3，Llama 4 Maverick，Qwen3-30b A3B，GPT-4O Mini和Llama 3.1 8B。我们的结果表明，简化通过忽略多义义，大大降低了定义完整性，从而增加了误解的风险。具有直接偏好优化的微调骆驼3.1 8b可大大提高所有及时类型的响应质量。这些发现凸显了需要平衡教育NLP的简单性和完整性，以确保所有学习者的可靠，背景感知的定义。

Title: Improving Data and Parameter Efficiency of Neural Language Models Using Representation Analysis

Authors: Josip Jukić
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.12004
Pdf URL: https://arxiv.org/pdf/2507.12004
Copy Paste: [[2507.12004]] Improving Data and Parameter Efficiency of Neural Language Models Using Representation Analysis(https://arxiv.org/abs/2507.12004)
Keywords: language model
Abstract: This thesis addresses challenges related to data and parameter efficiency in neural language models, with a focus on representation analysis and the introduction of new optimization techniques. The first part examines the properties and dynamics of language representations within neural models, emphasizing their significance in enhancing robustness and generalization. It proposes innovative approaches based on representation smoothness, including regularization strategies that utilize Jacobian and Hessian matrices to stabilize training and mitigate sensitivity to input perturbations. The second part focuses on methods to significantly enhance data and parameter efficiency by integrating active learning strategies with parameter-efficient fine-tuning, guided by insights from representation smoothness analysis. It presents smoothness-informed early-stopping techniques designed to eliminate the need for labeled validation sets and proposes innovative combinations of active learning and parameter-efficient fine-tuning to reduce labeling efforts and computational resources. Extensive experimental evaluations across various NLP tasks demonstrate that these combined approaches substantially outperform traditional methods in terms of performance, stability, and efficiency. The third part explores weak supervision techniques enhanced by in-context learning to effectively utilize unlabeled data, further reducing dependence on extensive labeling. It shows that using in-context learning as a mechanism for weak supervision enables models to better generalize from limited labeled data by leveraging unlabeled examples more effectively during training. Comprehensive empirical evaluations confirm significant gains in model accuracy, adaptability, and robustness, especially in low-resource settings and dynamic data environments.
摘要：本文解决了与神经语言模型中数据和参数效率相关的挑战，重点是表示分析和引入新的优化技术。第一部分研究了神经模型中语言表示的属性和动态，并强调了它们在增强鲁棒性和概括方面的意义。它提出了基于表示平滑度的创新方法，包括利用Jacobian和Hessian矩阵来稳定训练并减轻对输入扰动的敏感性的正规化策略。第二部分的重点是通过将主动学习策略与参数有效的微调整合在一起，以显着提高数据和参数效率的方法，并在表示平滑度分析中的见解为指导下。它提出了平稳性信息的早期技术技术，旨在消除对标记验证集的需求，并提出了积极学习和参数有效的微调的创新组合，以减少标签工作和计算资源。各种NLP任务的广泛实验评估表明，这些合并方法在性能，稳定性和效率方面大大优于传统方法。第三部分探讨了通过内在的学习来增强的弱监督技术，以有效地利用未标记的数据，从而进一步降低了对广泛标签的依赖。它表明，使用秘密学习作为弱监督的机制，可以通过在训练过程中更有效地利用未标记的示例来更好地从有限的标记数据中概括。全面的经验评估证实了模型准确性，适应性和鲁棒性的显着提高，尤其是在低资源设置和动态数据环境中。

Title: A Comparative Approach to Assessing Linguistic Creativity of Large Language Models and Humans

Authors: Anca Dinu, Andra-Maria Florescu, Alina Resceanu
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.12039
Pdf URL: https://arxiv.org/pdf/2507.12039
Copy Paste: [[2507.12039]] A Comparative Approach to Assessing Linguistic Creativity of Large Language Models and Humans(https://arxiv.org/abs/2507.12039)
Keywords: language model, llm
Abstract: The following paper introduces a general linguistic creativity test for humans and Large Language Models (LLMs). The test consists of various tasks aimed at assessing their ability to generate new original words and phrases based on word formation processes (derivation and compounding) and on metaphorical language use. We administered the test to 24 humans and to an equal number of LLMs, and we automatically evaluated their answers using OCSAI tool for three criteria: Originality, Elaboration, and Flexibility. The results show that LLMs not only outperformed humans in all the assessed criteria, but did better in six out of the eight test tasks. We then computed the uniqueness of the individual answers, which showed some minor differences between humans and LLMs. Finally, we performed a short manual analysis of the dataset, which revealed that humans are more inclined towards E(extending)-creativity, while LLMs favor F(ixed)-creativity.
摘要：以下论文介绍了针对人类和大型语言模型（LLM）的一般语言创造力测试。该测试包括各种任务，旨在评估其基于单词形成过程（派生和复合）以及隐喻语言使用的新原始单词和短语的能力。我们对24人进行了测试和相等数量的LLM，并使用OCSAI工具自动评估了他们的答案，以达到三个标准：原创性，阐述和灵活性。结果表明，LLM不仅在所有评估的标准中都超过了人类的表现，而且在八个测试任务中的六项中表现更好。然后，我们计算了单个答案的唯一性，这显示了人类和LLM之间的一些较小差异。最后，我们对数据集进行了简短的手动分析，该分析表明，人类更倾向于E（扩展）创造性，而LLMS则偏爱F（IXED）创造性。

Title: Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited

Authors: Anthony G Cohn, Robert E Blackwell
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.12059
Pdf URL: https://arxiv.org/pdf/2507.12059
Copy Paste: [[2507.12059]] Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited(https://arxiv.org/abs/2507.12059)
Keywords: language model, llm, agent
Abstract: We investigate the abilities of 28 Large language Models (LLMs) to reason about cardinal directions (CDs) using a benchmark generated from a set of templates, extensively testing an LLM's ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first, second or third person. Even the newer Large Reasoning Models are unable to reliably determine the correct CD for all questions. This paper summarises and extends earlier work presented at COSIT-24.
摘要：我们使用从一组模板中生成的基准来研究28个大语言模型（LLM）的能力，以推理基准方向（CDS），从而广泛测试了LLM确定特定情况下正确CD的能力。这些模板允许多个变化程度，例如所涉及的代理的运动均值，以及在第一，第二，第三人称中设置。即使是较新的大型推理模型也无法可靠地确定所有问题的正确CD。本文总结并扩展了COSIT-24上介绍的早期工作。

Title: Findings of MEGA: Maths Explanation with LLMs using the Socratic Method for Active Learning

Authors: Tosin Adewumi, Foteini Simistira Liwicki, Marcus Liwicki, Viktor Gardelli, Lama Alkhaled, Hamam Mokayed
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.12079
Pdf URL: https://arxiv.org/pdf/2507.12079
Copy Paste: [[2507.12079]] Findings of MEGA: Maths Explanation with LLMs using the Socratic Method for Active Learning(https://arxiv.org/abs/2507.12079)
Keywords: language model, gpt, llm
Abstract: This paper presents an intervention study on the effects of the combined methods of (1) the Socratic method, (2) Chain of Thought (CoT) reasoning, (3) simplified gamification and (4) formative feedback on university students' Maths learning driven by large language models (LLMs). We call our approach Mathematics Explanations through Games by AI LLMs (MEGA). Some students struggle with Maths and as a result avoid Math-related discipline or subjects despite the importance of Maths across many fields, including signal processing. Oftentimes, students' Maths difficulties stem from suboptimal pedagogy. We compared the MEGA method to the traditional step-by-step (CoT) method to ascertain which is better by using a within-group design after randomly assigning questions for the participants, who are university students. Samples (n=60) were randomly drawn from each of the two test sets of the Grade School Math 8K (GSM8K) and Mathematics Aptitude Test of Heuristics (MATH) datasets, based on the error margin of 11%, the confidence level of 90%, and a manageable number of samples for the student evaluators. These samples were used to evaluate two capable LLMs at length (Generative Pretrained Transformer 4o (GPT4o) and Claude 3.5 Sonnet) out of the initial six that were tested for capability. The results showed that students agree in more instances that the MEGA method is experienced as better for learning for both datasets. It is even much better than the CoT (47.5% compared to 26.67%) in the more difficult MATH dataset, indicating that MEGA is better at explaining difficult Maths problems.
摘要：本文介绍了一项干预研究（（1）苏格拉底式方法，（2）思想链（COT）推理，（3）简化游戏化和（4）对大学生数学学习的形成反馈（LLMS）的形成反馈。我们通过AI LLM（Mega）的游戏称呼我们的方法数学解释。一些学生在数学上挣扎，结果避免了与数学相关的纪律或受试者，尽管在许多领域（包括信号处理）中数学的重要性。通常，学生的数学困难源于次优教学法。我们将Mega方法与传统的逐步（COT）方法进行了比较，以确定哪种方法是通过随机为大学生的参与者随机分配问题后使用组内设计更好的。基于11％的误差率，置信度为90％，以及学生评估者的样本数量，从启发式方法（MATH）数据集的两个测试集中随机绘制了样品（n = 60），并从两个测试集中的每个测试集和数学能力测试。这些样品用于评估两个在最初的六个能力测试的最初六个功能的LLM（生成预验证的变压器4O（GPT4O）和Claude 3.5十四行诗）中。结果表明，学生在越来越多的情况下同意，大型方法对两个数据集的学习都更好。在更困难的数学数据集中，它甚至比COT好得多（47.5％，而为26.67％），这表明大型胜地更好地解释了困难的数学问题。

Title: Iterative Augmentation with Summarization Refinement (IASR) Evaluation for Unstructured Survey data Modeling and Analysis

Authors: Payal Bhattad, Sai Manoj Pudukotai Dinakarrao, Anju Gupta
Subjects: cs.CL, cs.LG
Abstract URL: https://arxiv.org/abs/2507.12126
Pdf URL: https://arxiv.org/pdf/2507.12126
Copy Paste: [[2507.12126]] Iterative Augmentation with Summarization Refinement (IASR) Evaluation for Unstructured Survey data Modeling and Analysis(https://arxiv.org/abs/2507.12126)
Keywords: language model, gpt, llm
Abstract: Text data augmentation is a widely used strategy for mitigating data sparsity in natural language processing (NLP), particularly in low-resource settings where limited samples hinder effective semantic modeling. While augmentation can improve input diversity and downstream interpretability, existing techniques often lack mechanisms to ensure semantic preservation during large-scale or iterative generation, leading to redundancy and instability. This work introduces a principled evaluation framework for large language model (LLM) based text augmentation, comprising two components: (1) Scalability Analysis, which measures semantic consistency as augmentation volume increases, and (2) Iterative Augmentation with Summarization Refinement (IASR), which evaluates semantic drift across recursive paraphrasing cycles. Empirical evaluations across state-of-the-art LLMs show that GPT-3.5 Turbo achieved the best balance of semantic fidelity, diversity, and generation efficiency. Applied to a real-world topic modeling task using BERTopic with GPT-enhanced few-shot labeling, the proposed approach results in a 400% increase in topic granularity and complete elimination of topic overlaps. These findings validated the utility of the proposed frameworks for structured evaluation of LLM-based augmentation in practical NLP pipelines.
摘要：文本数据增强是一种广泛使用的策略，用于减轻自然语言处理（NLP）中的数据稀疏性，尤其是在有限样本阻碍有效语义建模的低资源环境中。尽管增强可以改善投入多样性和下游解释性，但现有技术通常缺乏确保在大规模或迭代产生期间语义保存的机制，从而导致冗余和不稳定。这项工作介绍了基于大语言模型（LLM）的原则评估框架，包括两个组成部分：（1）可伸缩性分析，该分析随着增强体积的增加而衡量语义一致性，（2）迭代增强及其摘要改进（IASR），评估了跨递归杂物的语义漂移。最先进的LLMS的经验评估表明，GPT-3.5 Turbo实现了语义忠诚，多样性和发电效率的最佳平衡。该建议的方法应用于现实世界中的主题建模任务，并使用bertopic和gpt增强的几杆标签，因此提出的方法导致主题粒度性增加了400％，并完全消除了主题重叠。这些发现验证了所提出的框架对实用NLP管道中基于LLM的增强的结构化评估的实用性。

Title: Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators

Authors: Pavel Šindelář, Ondřej Bojar
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.12143
Pdf URL: https://arxiv.org/pdf/2507.12143
Copy Paste: [[2507.12143]] Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators(https://arxiv.org/abs/2507.12143)
Keywords: language model, llm
Abstract: ELOQUENT is a set of shared tasks that aims to create easily testable high-level criteria for evaluating generative language models. Sensemaking is one such shared task. In Sensemaking, we try to assess how well generative models ``make sense out of a given text'' in three steps inspired by exams in a classroom setting: (1) Teacher systems should prepare a set of questions, (2) Student systems should answer these questions, and (3) Evaluator systems should score these answers, all adhering rather strictly to a given set of input materials. We report on the 2025 edition of Sensemaking, where we had 7 sources of test materials (fact-checking analyses of statements, textbooks, transcribed recordings of a lecture, and educational videos) spanning English, German, Ukrainian, and Czech languages. This year, 4 teams participated, providing us with 2 Teacher submissions, 2 Student submissions, and 2 Evaluator submissions. We added baselines for Teacher and Student using commercial large language model systems. We devised a fully automatic evaluation procedure, which we compare to a minimalistic manual evaluation. We were able to make some interesting observations. For the first task, the creation of questions, better evaluation strategies will still have to be devised because it is difficult to discern the quality of the various candidate question sets. In the second task, question answering, the LLMs examined overall perform acceptably, but restricting their answers to the given input texts remains problematic. In the third task, evaluation of question answers, our adversarial tests reveal that systems using the LLM-as-a-Judge paradigm erroneously rate both garbled question-answer pairs and answers to mixed-up questions as acceptable.
摘要：雄辩是一组共享任务，旨在创建易于测试的高级标准来评估生成语言模型。感官就是这样的共同任务。在感官中，我们试图评估生成模型在教室环境中受考试启发的三个步骤``从给定的文本中变得有意义''：（1）教师系统应准备一组问题，（2）学生系统应回答这些问题，（3）评估者系统应为这些答案进行评分，所有这些答案都相当严格地构成了给定的输入材料集合。我们报告了2025年的Sensemaking版本，其中有7种测试材料来源（对语句，教科书，讲座的抄录录音和教育视频的事实检查分析），涵盖英语，德语，乌克兰语和捷克语。今年，有4个团队参加了会议，为我们提供了2个教师提交，2个学生提交和2份评估员提交。我们使用商业大型语言模型系统为教师和学生添加了基线。我们设计了一个全自动评估程序，我们将其与简约的手动评估进行了比较。我们能够做出一些有趣的观察。对于第一个任务，提出问题的创建，仍然必须设计更好的评估策略，因为很难辨别各种候选问题集的质量。在第二个任务中，问答，LLMS检查了总体可接受的表现，但是将其答案限制在给定的输入文本中仍然有问题。在第三个任务中，对问题答案的评估，我们的对抗性测试表明，使用LLM-AS-A-A-Gudge范式的系统错误地对两个乱码的问答式汇编进行了评分，以及对混合问题的答案，可以接受。

Title: Toward a Behavioural Translation Style Space: Simulating the Temporal Dynamics of Affect, Behaviour, and Cognition in Human Translation Production

Authors: Michael Carl, Takanori Mizowaki, Aishvarya Ray, Masaru Yamada, Devi Sri Bandaru, Xinyue Ren
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.12208
Pdf URL: https://arxiv.org/pdf/2507.12208
Copy Paste: [[2507.12208]] Toward a Behavioural Translation Style Space: Simulating the Temporal Dynamics of Affect, Behaviour, and Cognition in Human Translation Production(https://arxiv.org/abs/2507.12208)
Keywords: agent
Abstract: The paper introduces a Behavioural Translation Style Space (BTSS) that describes possible behavioural translation patterns. The suggested BTSS is organized as a hierarchical structure that entails various embedded processing layers. We posit that observable translation behaviour - i.e., eye and finger movements - is fundamental when executing the physical act of translation but it is caused and shaped by higher-order cognitive processes and affective translation states. We analyse records of keystrokes and gaze data as indicators of the hidden mental processing structure and organize the behavioural patterns as a multi-layered embedded BTSS. The BTSS serves as the basis for a computational translation agent to simulate the temporal dynamics of affect, automatized behaviour and cognition during human translation production.
摘要：本文介绍了一个行为翻译样式空间（BTSS），该空间描述了可能的行为翻译模式。建议的BTSS是一种层次结构，需要各种嵌入式处理层。我们认为，在执行翻译的物理行为时，可观察到的翻译行为（即眼睛和手指运动）是基本的，但它是由高阶认知过程和情感翻译状态引起和塑造的。我们将击键和凝视数据的记录分析为隐藏的心理处理结构的指标，并将行为模式组织为多层嵌入式BTSS。 BTSS是模拟人类翻译生产过程中情感，自动化行为和认知的时间动态的计算翻译代理的基础。

Title: Improving Contextual ASR via Multi-grained Fusion with Large Language Models

Authors: Shilin Zhou, Zhenghua Li
Subjects: cs.CL, cs.AI
Abstract URL: https://arxiv.org/abs/2507.12252
Pdf URL: https://arxiv.org/pdf/2507.12252
Copy Paste: [[2507.12252]] Improving Contextual ASR via Multi-grained Fusion with Large Language Models(https://arxiv.org/abs/2507.12252)
Keywords: language model, llm
Abstract: While end-to-end Automatic Speech Recognition (ASR) models have shown impressive performance in transcribing general speech, they often struggle to accurately recognize contextually relevant keywords, such as proper nouns or user-specific entities. Previous approaches have explored leveraging keyword dictionaries in the textual modality to improve keyword recognition, either through token-level fusion that guides token-by-token generation or phrase-level fusion that enables direct copying of keyword phrases. However, these methods operate at different granularities and have their own limitations. In this paper, we propose a novel multi-grained fusion approach that jointly leverages the strengths of both token-level and phrase-level fusion with Large Language Models (LLMs). Our approach incorporates a late-fusion strategy that elegantly combines ASR's acoustic information with LLM's rich contextual knowledge, balancing fine-grained token precision with holistic phrase-level understanding. Experiments on Chinese and English datasets demonstrate that our approach achieves state-of-the-art performance on keyword-related metrics while preserving high accuracy on non-keyword text. Ablation studies further confirm that the token-level and phrase-level components both contribute significantly to the performance gains, complementing each other in our joint multi-grained framework. The code and models will be publicly available at this https URL.
摘要：虽然端到端的自动语音识别（ASR）模型在转录一般语音时表现出了令人印象深刻的性能，但他们经常难以准确识别上下文相关的关键字，例如专有名词或特定用户特定的实体。先前的方法已经通过文本方式探索了关键字字典来改善关键字识别，要么通过令牌级融合，该融合指导令牌逐字的生成或短语级融合，可以直接复制关键字短语。但是，这些方法以不同的粒度运行，并且有其自身的局限性。在本文中，我们提出了一种新型的多元融合方法，该方法共同利用令牌级别和短语级融合与大语言模型（LLMS）的优势。我们的方法结合了一种晚期融合策略，该策略将ASR的声学信息与LLM丰富的上下文知识相结合，将精细的令牌精度与整体短语级别的理解保持平衡。中文和英语数据集的实验表明，我们的方法在与关键字相关的指标上实现了最先进的性能，同时保留了非关键词文本的高精度。消融研究进一步证实，令牌级别和短语级成分都对性能提高产生了重大贡献，在我们的联合多元框架框架中相互补充。代码和模型将在此HTTPS URL上公开可用。

Title: Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese

Authors: Yikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.12260
Pdf URL: https://arxiv.org/pdf/2507.12260
Copy Paste: [[2507.12260]] Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese(https://arxiv.org/abs/2507.12260)
Keywords: language model
Abstract: In this paper, we propose the first quantitative measure for translationese -- the translationese-index (T-index) for graded and generalizable measurement of translationese, computed from the likelihood ratios of two contrastively fine-tuned language models (LMs). We use a synthesized dataset and a dataset with translations in the wild to evaluate T-index's generalizability in cross-domain settings and its validity against human judgments. Our results show that T-index is both robust and efficient. T-index scored by two 0.5B LMs fine-tuned on only 1-5k pairs of synthetic data can well capture translationese in the wild. We find that the relative differences in T-indices between translations can well predict pairwise translationese annotations obtained from human annotators; and the absolute values of T-indices correlate well with human ratings of degrees of translationese (Pearson's $r = 0.568$). Additionally, the correlation between T-index and existing machine translation (MT) quality estimation (QE) metrics such as BLEU and COMET is low, suggesting that T-index is not covered by these metrics and can serve as a complementary metric in MT QE.
摘要：在本文中，我们提出了翻译人员的第一个定量措施-TranslationEse-index（T-Index（T-Index），用于对翻译的分级和可推广测量，该测量是根据两个对比性微调语言模型（LMS）计算得出的。我们使用一个合成的数据集和一个带有野外翻译的数据集来评估T-Index在跨域设置中的普遍性及其针对人类判断的有效性。我们的结果表明，T索引既有稳定又有效。仅在1-5k对的合成数据上微调的两个0.5B LMS得分的T指数可以很好地捕获野外的翻译。我们发现，翻译之间的T指标的相对差异可以很好地预测从人类注释者获得的成对翻译注释。 T-Indices的绝对值与人类翻译学位的评分良好（Pearson的$ r = 0.568 $）。此外，T索引与现有的机器翻译（MT）质量估计（QE）指标（例如BLEU和COMET）之间的相关性很低，这表明这些指标不涵盖T-Index，并且可以用作MT QE中的互补度量。

Title: Infherno: End-to-end Agent-based FHIR Resource Synthesis from Free-form Clinical Notes

Authors: Johann Frei, Nils Feldhus, Lisa Raithel, Roland Roller, Alexander Meyer, Frank Kramer
Subjects: cs.CL, cs.AI
Abstract URL: https://arxiv.org/abs/2507.12261
Pdf URL: https://arxiv.org/pdf/2507.12261
Copy Paste: [[2507.12261]] Infherno: End-to-end Agent-based FHIR Resource Synthesis from Free-form Clinical Notes(https://arxiv.org/abs/2507.12261)
Keywords: llm, agent
Abstract: For clinical data integration and healthcare services, the HL7 FHIR standard has established itself as a desirable format for interoperability between complex health data. Previous attempts at automating the translation from free-form clinical notes into structured FHIR resources rely on modular, rule-based systems or LLMs with instruction tuning and constrained decoding. Since they frequently suffer from limited generalizability and structural inconformity, we propose an end-to-end framework powered by LLM agents, code execution, and healthcare terminology database tools to address these issues. Our solution, called Infherno, is designed to adhere to the FHIR document schema and competes well with a human baseline in predicting FHIR resources from unstructured text. The implementation features a front end for custom and synthetic data and both local and proprietary models, supporting clinical data integration processes and interoperability across institutions.
摘要：对于临床数据集成和医疗保健服务，HL7 FHIR标准已将自己确立为复杂健康数据之间互操作性的理想格式。以前的尝试将自由形式临床注释转换为结构化FHIR资源的翻译依赖于模块化，基于规则的系统或LLM，并使用指令调整和约束解码。由于它们经常遭受有限的通用性和结构性不合格性，因此我们提出了一个由LLM代理，代码执行和医疗保健术语数据库工具驱动的端到端框架来解决这些问题。我们的解决方案称为Infherno，旨在遵守FHIR文档模式，并与人类基线竞争，以预测非结构化文本的FHIR资源。该实现具有定制和综合数据以及本地和专有模型的前端，支持临床数据集成过程以及跨机构的互操作性。

Title: Text-ADBench: Text Anomaly Detection Benchmark based on LLMs Embedding

Authors: Feng Xiao, Jicong Fan
Subjects: cs.CL, cs.AI, cs.LG
Abstract URL: https://arxiv.org/abs/2507.12295
Pdf URL: https://arxiv.org/pdf/2507.12295
Copy Paste: [[2507.12295]] Text-ADBench: Text Anomaly Detection Benchmark based on LLMs Embedding(https://arxiv.org/abs/2507.12295)
Keywords: language model, llm
Abstract: Text anomaly detection is a critical task in natural language processing (NLP), with applications spanning fraud detection, misinformation identification, spam detection and content moderation, etc. Despite significant advances in large language models (LLMs) and anomaly detection algorithms, the absence of standardized and comprehensive benchmarks for evaluating the existing anomaly detection methods on text data limits rigorous comparison and development of innovative approaches. This work performs a comprehensive empirical study and introduces a benchmark for text anomaly detection, leveraging embeddings from diverse pre-trained language models across a wide array of text datasets. Our work systematically evaluates the effectiveness of embedding-based text anomaly detection by incorporating (1) early language models (GloVe, BERT); (2) multiple LLMs (LLaMa-2, LLama-3, Mistral, OpenAI (small, ada, large)); (3) multi-domain text datasets (news, social media, scientific publications); (4) comprehensive evaluation metrics (AUROC, AUPRC). Our experiments reveal a critical empirical insight: embedding quality significantly governs anomaly detection efficacy, and deep learning-based approaches demonstrate no performance advantage over conventional shallow algorithms (e.g., KNN, Isolation Forest) when leveraging LLM-derived this http URL addition, we observe strongly low-rank characteristics in cross-model performance matrices, which enables an efficient strategy for rapid model evaluation (or embedding evaluation) and selection in practical applications. Furthermore, by open-sourcing our benchmark toolkit that includes all embeddings from different models and code at this https URL, this work provides a foundation for future research in robust and scalable text anomaly detection systems.
摘要：文本异常检测是自然语言处理（NLP）的一项关键任务，涵盖欺诈检测，错误信息识别，垃圾邮件检测和内容节制等，尽管大型语言模型（LLMS）和异常检测算法取得了重大进展，并且缺乏标准化和全面的基本标记，并且可以评估现有的探测范围的图形数据和综合基准。这项工作执行了一项全面的实证研究，并介绍了文本异常检测的基准，并利用各种文本数据集中利用各种预训练的语言模型的嵌入。我们的工作系统地评估了基于（1）早期语言模型（Glove，Bert）的基于嵌入的文本异常检测的有效性；（2）多个LLM（Llama-2，Llama-3，Mismtral，Openai（Small，Ada，grount））; （3）多域文本数据集（新闻，社交媒体，科学出版物）；（4）全面评估指标（AUROC，AUPRC）。 Our experiments reveal a critical empirical insight: embedding quality significantly governs anomaly detection efficacy, and deep learning-based approaches demonstrate no performance advantage over conventional shallow algorithms (e.g., KNN, Isolation Forest) when leveraging LLM-derived this http URL addition, we observe strongly low-rank characteristics in cross-model performance matrices, which enables an efficient strategy for rapid model evaluation (or嵌入评估）和在实际应用中进行选择。此外，通过开源我们的基准工具包，该工具包包括此HTTPS URL中不同模型和代码的所有嵌入，这项工作为未来的研究中的研究为可靠，可扩展的文本异常检测系统提供了基础。

Title: Chain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarization

Authors: Prashanth Vijayaraghavan, Apoorva Nitsure, Charles Mackin, Luyao Shi, Stefano Ambrogio, Arvind Haran, Viresh Paruthi, Ali Elzein, Dan Coops, David Beymer, Tyler Baldwin, Ehsan Degan
Subjects: cs.CL, cs.AI, cs.AR
Abstract URL: https://arxiv.org/abs/2507.12308
Pdf URL: https://arxiv.org/pdf/2507.12308
Copy Paste: [[2507.12308]] Chain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarization(https://arxiv.org/abs/2507.12308)
Keywords: language model, llm, prompt
Abstract: Large Language Models (LLMs) have become widely used across diverse NLP tasks and domains, demonstrating their adaptability and effectiveness. In the realm of Electronic Design Automation (EDA), LLMs show promise for tasks like Register-Transfer Level (RTL) code generation and summarization. However, despite the proliferation of LLMs for general code-related tasks, there's a dearth of research focused on evaluating and refining these models for hardware description languages (HDLs), notably VHDL. In this study, we evaluate the performance of existing code LLMs for VHDL code generation and summarization using various metrics and two datasets -- VHDL-Eval and VHDL-Xform. The latter, an in-house dataset, aims to gauge LLMs' understanding of functionally equivalent code. Our findings reveal consistent underperformance of these models across different metrics, underscoring a significant gap in their suitability for this domain. To address this challenge, we propose Chain-of-Descriptions (CoDes), a novel approach to enhance the performance of LLMs for VHDL code generation and summarization tasks. CoDes involves generating a series of intermediate descriptive steps based on: (i) the problem statement for code generation, and (ii) the VHDL code for summarization. These steps are then integrated with the original input prompt (problem statement or code) and provided as input to the LLMs to generate the final output. Our experiments demonstrate that the CoDes approach significantly surpasses the standard prompting strategy across various metrics on both datasets. This method not only improves the quality of VHDL code generation and summarization but also serves as a framework for future research aimed at enhancing code LLMs for VHDL.
摘要：大型语言模型（LLMS）已广泛用于不同的NLP任务和领域，以证明其适应性和有效性。在电子设计自动化领域（EDA）中，LLMS显示了诸如寄存器转移级别（RTL）代码生成和摘要之类的任务的希望。然而，尽管LLM对一般代码相关的任务的扩散，但缺乏研究重点是评估和完善这些模型的硬件说明语言（HDLS），尤其是VHDL。在这项研究中，我们使用各种指标和两个数据集评估了VHDL代码生成的现有代码LLMS的性能 - VHDL-EVAL和VHDL-XFORM。后者是内部数据集，旨在评估LLMS对功能等效代码的理解。我们的发现揭示了这些模型在不同指标上的表现不佳，这突显了其适合该领域的显着差距。为了应对这一挑战，我们提出了描述链（代码），这是一种增强LLMS vhdl代码生成和摘要任务的新方法。代码涉及基于：（i）代码生成的问题语句以及（ii）汇总的VHDL代码。然后将这些步骤与原始输入提示（问题语句或代码）集成在一起，并作为LLMS提供的输入以生成最终输出。我们的实验表明，代码方法可显着超过两个数据集上各种指标的标准提示策略。该方法不仅可以提高VHDL代码生成和摘要的质量，而且还可以作为未来研究的框架，旨在增强VHDL代码LLMS。

Title: Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debate

Authors: Ana Davila, Jacinto Colan, Yasuhisa Hasegawa
Subjects: cs.CL, cs.HC
Abstract URL: https://arxiv.org/abs/2507.12370
Pdf URL: https://arxiv.org/pdf/2507.12370
Copy Paste: [[2507.12370]] Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debate(https://arxiv.org/abs/2507.12370)
Keywords: language model, llm, agent
Abstract: Large Language Models (LLMs) have demonstrated significant capabilities in understanding and generating human language, contributing to more natural interactions with complex systems. However, they face challenges such as ambiguity in user requests processed by LLMs. To address these challenges, this paper introduces and evaluates a multi-agent debate framework designed to enhance detection and resolution capabilities beyond single models. The framework consists of three LLM architectures (Llama3-8B, Gemma2-9B, and Mistral-7B variants) and a dataset with diverse ambiguities. The debate framework markedly enhanced the performance of Llama3-8B and Mistral-7B variants over their individual baselines, with Mistral-7B-led debates achieving a notable 76.7% success rate and proving particularly effective for complex ambiguities and efficient consensus. While acknowledging varying model responses to collaborative strategies, these findings underscore the debate framework's value as a targeted method for augmenting LLM capabilities. This work offers important insights for developing more robust and adaptive language understanding systems by showing how structured debates can lead to improved clarity in interactive systems.
摘要：大型语言模型（LLM）在理解和生成人类语言方面表现出了重要的能力，这有助于与复杂系统更自然的互动。 However, they face challenges such as ambiguity in user requests processed by LLMs.为了应对这些挑战，本文介绍并评估了一个多代理辩论框架，旨在增强单个模型以外的检测和分辨率。 The framework consists of three LLM architectures (Llama3-8B, Gemma2-9B, and Mistral-7B variants) and a dataset with diverse ambiguities.辩论框架明显提高了Llama3-8b和Mismtral-7b在各个基本线上的表现，Mistral-7B领导的辩论取得了显着的76.7％的成功率，并证明了对复杂的歧义和有效共识的特别有效。在承认对协作策略的不同模型响应的同时，这些发现强调了辩论框架作为增强LLM功能的目标方法的价值。这项工作提供了重要的见解，可以通过展示结构化辩论如何改善交互式系统的清晰度来开发更强大和自适应的语言理解系统。

Title: Web-Browsing LLMs Can Access Social Media Profiles and Infer User Demographics

Authors: Meysam Alizadeh, Fabrizio Gilardi, Zeynab Samei, Mohsen Mosleh
Subjects: cs.CL
Abstract URL: https://arxiv.org/abs/2507.12372
Pdf URL: https://arxiv.org/pdf/2507.12372
Copy Paste: [[2507.12372]] Web-Browsing LLMs Can Access Social Media Profiles and Infer User Demographics(https://arxiv.org/abs/2507.12372)
Keywords: language model, llm
Abstract: Large language models (LLMs) have traditionally relied on static training data, limiting their knowledge to fixed snapshots. Recent advancements, however, have equipped LLMs with web browsing capabilities, enabling real time information retrieval and multi step reasoning over live web content. While prior studies have demonstrated LLMs ability to access and analyze websites, their capacity to directly retrieve and analyze social media data remains unexplored. Here, we evaluate whether web browsing LLMs can infer demographic attributes of social media users given only their usernames. Using a synthetic dataset of 48 X (Twitter) accounts and a survey dataset of 1,384 international participants, we show that these models can access social media content and predict user demographics with reasonable accuracy. Analysis of the synthetic dataset further reveals how LLMs parse and interpret social media profiles, which may introduce gender and political biases against accounts with minimal activity. While this capability holds promise for computational social science in the post API era, it also raises risks of misuse particularly in information operations and targeted advertising underscoring the need for safeguards. We recommend that LLM providers restrict this capability in public facing applications, while preserving controlled access for verified research purposes.
摘要：传统上，大型语言模型（LLMS）依靠静态培训数据，将知识限制为固定快照。但是，最近的进步已将LLMS配备了Web浏览功能，从而使实时信息检索和多步骤推理可以超越实时Web内容。虽然先前的研究表明LLM可以访问和分析网站的能力，但他们直接检索和分析社交媒体数据的能力仍未得到探索。在这里，我们评估网络浏览LLM是否可以推断社交媒体用户的人口统计属性仅给定其用户名。使用48 x（Twitter）帐户的合成数据集和1,384名国际参与者的调查数据集，我们表明这些模型可以访问社交媒体内容并以合理的准确性来预测用户人口统计信息。对合成数据集的分析进一步揭示了LLMS解析和解释社交媒体概况，这可能会引入针对最少活动的账目的性别和政治偏见。尽管此能力在API后时代有望对计算社会科学有望，但它也会增加滥用的风险，尤其是在信息操作和有针对性的广告中，强调了对保障措施的需求。我们建议LLM提供商在公共面对应用程序中限制此功能，同时保留受控访问以进行验证的研究目的。

Title: Probing for Arithmetic Errors in Language Models

Authors: Yucheng Sun, Alessandro Stolfo, Mrinmaya Sachan
Subjects: cs.CL, cs.AI
Abstract URL: https://arxiv.org/abs/2507.12379
Pdf URL: https://arxiv.org/pdf/2507.12379
Copy Paste: [[2507.12379]] Probing for Arithmetic Errors in Language Models(https://arxiv.org/abs/2507.12379)
Keywords: language model, prompt, chain-of-thought
Abstract: We investigate whether internal activations in language models can be used to detect arithmetic errors. Starting with a controlled setting of 3-digit addition, we show that simple probes can accurately decode both the model's predicted output and the correct answer from hidden states, regardless of whether the model's output is correct. Building on this, we train lightweight error detectors that predict model correctness with over 90% accuracy. We then extend our analysis to structured chain-of-thought traces on addition-only GSM8K problems and find that probes trained on simple arithmetic generalize well to this more complex setting, revealing consistent internal representations. Finally, we demonstrate that these probes can guide selective re-prompting of erroneous reasoning steps, improving task accuracy with minimal disruption to correct outputs. Our findings suggest that arithmetic errors can be anticipated from internal activations alone, and that simple probes offer a viable path toward lightweight model self-correction.
摘要：我们研究语言模型中的内部激活是否可以用于检测算术错误。从3位添加的受控设置开始，我们表明，无论模型的输出是否正确，简单的探针都可以准确地解码模型的预测输出和正确的答案。在此基础上，我们训练轻巧的错误检测器，这些检测器可预测模型的正确性，精度超过90％。然后，我们将分析扩展到仅加成的GSM8K问题上的结构化链痕迹，并发现对简单算术训练的探针很好地推广到更复杂的设置，从而揭示了一致的内部表示。最后，我们证明这些探针可以指导选择性重新提出错误的推理步骤，从而通过最小的破坏提高任务准确性以纠正输出。我们的发现表明，仅凭内部激活就可以预见算术错误，而简单的探针为轻质模型自校正提供了可行的途径。

Title: Advancing Retrieval-Augmented Generation for Structured Enterprise and Internal Data

Authors: Chandana Cheerla
Subjects: cs.CL, cs.AI, cs.CE, cs.IR
Abstract URL: https://arxiv.org/abs/2507.12425
Pdf URL: https://arxiv.org/pdf/2507.12425
Copy Paste: [[2507.12425]] Advancing Retrieval-Augmented Generation for Structured Enterprise and Internal Data(https://arxiv.org/abs/2507.12425)
Keywords: language model, llm, retrieval-augmented generation, agent
Abstract: Organizations increasingly rely on proprietary enterprise data, including HR records, structured reports, and tabular documents, for critical decision-making. While Large Language Models (LLMs) have strong generative capabilities, they are limited by static pretraining, short context windows, and challenges in processing heterogeneous data formats. Conventional Retrieval-Augmented Generation (RAG) frameworks address some of these gaps but often struggle with structured and semi-structured data. This work proposes an advanced RAG framework that combines hybrid retrieval strategies using dense embeddings (all-mpnet-base-v2) and BM25, enhanced by metadata-aware filtering with SpaCy NER and cross-encoder reranking. The framework applies semantic chunking to maintain textual coherence and retains tabular data structures to preserve row-column integrity. Quantized indexing optimizes retrieval efficiency, while human-in-the-loop feedback and conversation memory improve adaptability. Experiments on enterprise datasets show notable improvements: Precision@5 increased by 15 percent (90 versus 75), Recall@5 by 13 percent (87 versus 74), and Mean Reciprocal Rank by 16 percent (0.85 versus 0.69). Qualitative evaluations show higher scores in Faithfulness (4.6 versus 3.0), Completeness (4.2 versus 2.5), and Relevance (4.5 versus 3.2) on a 5-point Likert scale. These results demonstrate the framework's effectiveness in delivering accurate, comprehensive, and contextually relevant responses for enterprise tasks. Future work includes extending to multimodal data and integrating agent-based retrieval. The source code will be released at this https URL
摘要：组织越来越依赖专有企业数据，包括人力资源记录，结构化报告和表格文件，以进行关键的决策。尽管大型语言模型（LLMS）具有强大的生成能力，但它们受到静态预处理，短上下文窗口以及处理异质数据格式的挑战的限制。常规的检索演示生成（RAG）框架解决了其中一些差距，但经常在结构化和半结构化数据上挣扎。这项工作提出了一个先进的抹布框架，该框架结合了使用密集的嵌入（All-Mpnet-Base-V2）和BM25结合混合检索策略，通过元数据感知过滤和Spacy NER和交叉编码器reranking增强了混合框架。该框架应用语义块来维持文本连贯性并保留表格数据结构以保持行柱完整性。量化的索引优化了检索效率，而人类的反馈和对话记忆可提高适应性。企业数据集上的实验表现出显着的改进：Precision@5增加了15％（90对75），召回@5升至13％（87对74），平均互惠等级提高16％（0.85对0.69）。定性评估显示，忠实分数（4.6对3.0），完整性（4.2对2.5）以及相关性（4.5对3.2），以5分李克特量表为单位。这些结果证明了该框架在为企业任务提供准确，全面和上下文相关的响应方面的有效性。未来的工作包括扩展到多模式数据并集成基于代理的检索。源代码将在此HTTPS URL上发布

Title: Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models

Authors: Yik Siu Chan, Zheng-Xin Yong, Stephen H. Bach
Subjects: cs.CL, cs.AI, cs.LG
Abstract URL: https://arxiv.org/abs/2507.12428
Pdf URL: https://arxiv.org/pdf/2507.12428
Copy Paste: [[2507.12428]] Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models(https://arxiv.org/abs/2507.12428)
Keywords: language model
Abstract: Open-weights reasoning language models generate long chains-of-thought (CoTs) before producing a final response, which improves performance but introduces additional alignment risks, with harmful content often appearing in both the CoTs and the final outputs. In this work, we investigate if we can use CoTs to predict final response misalignment. We evaluate a range of monitoring approaches, including humans, highly-capable large language models, and text classifiers, using either CoT text or activations. First, we find that a simple linear probe trained on CoT activations can significantly outperform all text-based methods in predicting whether a final response will be safe or unsafe. CoT texts are often unfaithful and can mislead humans and classifiers, while model latents (i.e., CoT activations) offer a more reliable predictive signal. Second, the probe makes accurate predictions before reasoning completes, achieving strong performance even when applied to early CoT segments. These findings generalize across model sizes, families, and safety benchmarks, suggesting that lightweight probes could enable real-time safety monitoring and early intervention during generation.
摘要：开放重量推理语言模型在产生最终响应之前会产生长链（COTS），从而提高了性能，但会引入额外的对齐风险，而COTS和最终输出中经常出现有害内容。 In this work, we investigate if we can use CoTs to predict final response misalignment.我们使用COT文本或激活评估包括人类，高度能力的大语言模型和文本分类器在内的一系列监视方法。首先，我们发现在COT激活上训练的简单线性探针可以显着优于所有基于文本的方法，以预测最终响应是安全还是不安全。 COT文本通常是不忠实的，并且会误导人和分类器，而模型潜在（即COT激活）提供了更可靠的预测信号。 Second, the probe makes accurate predictions before reasoning completes, achieving strong performance even when applied to early CoT segments.这些发现跨越了模型尺寸，家庭和安全基准，表明轻量级探针可以实时安全监控和发电期间的早期干预。

Title: Language Models Improve When Pretraining Data Matches Target Tasks

Authors: David Mizrahi, Anders Boesen Lindbo Larsen, Jesse Allardice, Suzie Petryk, Yuri Gorokhov, Jeffrey Li, Alex Fang, Josh Gardner, Tom Gunter, Afshin Dehghan
Subjects: cs.CL, cs.LG
Abstract URL: https://arxiv.org/abs/2507.12466
Pdf URL: https://arxiv.org/pdf/2507.12466
Copy Paste: [[2507.12466]] Language Models Improve When Pretraining Data Matches Target Tasks(https://arxiv.org/abs/2507.12466)
Keywords: language model
Abstract: Every data selection method inherently has a target. In practice, these targets often emerge implicitly through benchmark-driven iteration: researchers develop selection strategies, train models, measure benchmark performance, then refine accordingly. This raises a natural question: what happens when we make this optimization explicit? To explore this, we propose benchmark-targeted ranking (BETR), a simple method that selects pretraining documents based on similarity to benchmark training examples. BETR embeds benchmark examples and a sample of pretraining documents in a shared space, scores this sample by similarity to benchmarks, then trains a lightweight classifier to predict these scores for the full corpus. We compare data selection methods by training over 500 models spanning $10^{19}$ to $10^{22}$ FLOPs and fitting scaling laws to them. From this, we find that simply aligning pretraining data to evaluation benchmarks using BETR achieves a 2.1x compute multiplier over DCLM-Baseline (4.7x over unfiltered data) and improves performance on 9 out of 10 tasks across all scales. BETR also generalizes well: when targeting a diverse set of benchmarks disjoint from our evaluation suite, it still matches or outperforms baselines. Our scaling analysis further reveals a clear trend: larger models require less aggressive filtering. Overall, our findings show that directly matching pretraining data to target tasks precisely shapes model capabilities and highlight that optimal selection strategies must adapt to model scale.
摘要：每种数据选择方法本质上都有一个目标。实际上，这些目标通常通过基准驱动的迭代隐式出现：研究人员制定了选择策略，火车模型，测量基准的性能，然后相应地完善。这提出了一个自然的问题：当我们明确地进行优化时会发生什么？为了探讨这一点，我们提出了以基准为目标的排名（BETR），这是一种基于与基准培训示例的相似性选择预训练文档的简单方法。 BETR在共享空间中嵌入基准示例和一个预处理的文档样本，通过与基准相似的方式得分此样本，然后训练轻质分类器，以预测完整语料库的这些分数。我们通过训练超过500个型号的$ 10^{19} $到$ 10^{22} $ flops和拟合缩放法则来比较数据选择方法。从中，我们发现，只需将预处理数据与BETR对准基准测试，可以在DCLM-Baseline上实现2.1倍的计算乘数（未经过滤的数据4.7倍），并在所有尺度上提高10个任务中的9个任务的性能。 BETR也很好地概括了：当针对与我们的评估套件的各种基准分配不相交时，它仍然与基准相匹配或胜过基线。我们的缩放分析进一步揭示了一个清晰的趋势：较大的模型需要较少的攻击性过滤。总体而言，我们的发现表明，直接匹配预处理数据与目标任务的匹配精确地塑造了模型功能，并强调了最佳选择策略必须适应模型量表。