2024-10-17

Title: A Robust Multisource Remote Sensing Image Matching Method Utilizing Attention and Feature Enhancement Against Noise Interference

• Authors: Yuan Li, Dapeng Wu, Yaping Cui, Peng He, Yuan Zhang, Ruyan Wang
• Subjects: cs.CV, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.11848
• Pdf URL: https://arxiv.org/pdf/2410.11848
• Copy Paste: [[2410.11848]] A Robust Multisource Remote Sensing Image Matching Method Utilizing Attention and Feature Enhancement Against Noise Interference(https://arxiv.org/abs/2410.11848)
• Keywords: robust, extraction, transformer
• Abstract: Image matching is a fundamental and critical task of multisource remote sensing image applications. However, remote sensing images are susceptible to various noises. Accordingly, how to effectively achieve accurate matching in noise images is a challenging problem. To solve this issue, we propose a robust multisource remote sensing image matching method utilizing attention and feature enhancement against noise interference. In the first stage, we combine deep convolution with the attention mechanism of transformer to perform dense feature extraction, constructing feature descriptors with higher discriminability and robustness. Subsequently, we employ a coarse-to-fine matching strategy to achieve dense matches. In the second stage, we introduce an outlier removal network based on a binary classification mechanism, which can establish effective and geometrically consistent correspondences between images; through weighting for each correspondence, inliers vs. outliers classification are performed, as well as removing outliers from dense matches. Ultimately, we can accomplish more efficient and accurate matches. To validate the performance of the proposed method, we conduct experiments using multisource remote sensing image datasets for comparison with other state-of-the-art methods under different scenarios, including noise-free, additive random noise, and periodic stripe noise. Comparative results indicate that the proposed method has a more well-balanced performance and robustness. The proposed method contributes a valuable reference for solving the difficult problem of noise image matching.

Title: Neural Metamorphosis

• Authors: Xingyi Yang, Xinchao Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11878
• Pdf URL: https://arxiv.org/pdf/2410.11878
• Copy Paste: [[2410.11878]] Neural Metamorphosis(https://arxiv.org/abs/2410.11878)
• Keywords: segmentation
• Abstract: This paper introduces a new learning paradigm termed Neural Metamorphosis (NeuMeta), which aims to build self-morphable neural networks. Contrary to crafting separate models for different architectures or sizes, NeuMeta directly learns the continuous weight manifold of neural networks. Once trained, we can sample weights for any-sized network directly from the manifold, even for previously unseen configurations, without retraining. To achieve this ambitious goal, NeuMeta trains neural implicit functions as hypernetworks. They accept coordinates within the model space as input, and generate corresponding weight values on the manifold. In other words, the implicit function is learned in a way, that the predicted weights is well-performed across various models sizes. In training those models, we notice that, the final performance closely relates on smoothness of the learned manifold. In pursuit of enhancing this smoothness, we employ two strategies. First, we permute weight matrices to achieve intra-model smoothness, by solving the Shortest Hamiltonian Path problem. Besides, we add a noise on the input coordinates when training the implicit function, ensuring models with various sizes shows consistent outputs. As such, NeuMeta shows promising results in synthesizing parameters for various network configurations. Our extensive tests in image classification, semantic segmentation, and image generation reveal that NeuMeta sustains full-size performance even at a 75% compression rate.

Title: Development and Testing of a Wood Panels Bark Removal Equipment Based on Deep Learning

• Authors: Rijun Wang, Guanghao Zhang, Hongyang Chen, Xinye Yu, Yesheng Chen, Fulong Liang, Xiangwei Mou, Bo Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11913
• Pdf URL: https://arxiv.org/pdf/2410.11913
• Copy Paste: [[2410.11913]] Development and Testing of a Wood Panels Bark Removal Equipment Based on Deep Learning(https://arxiv.org/abs/2410.11913)
• Keywords: segmentation
• Abstract: Attempting to apply deep learning methods to wood panels bark removal equipment to enhance the quality and efficiency of bark removal is a significant and challenging endeavor. This study develops and tests a deep learning-based wood panels bark removal equipment. In accordance with the practical requirements of sawmills, a wood panels bark removal equipment equipped with a vision inspection system is designed. Based on a substantial collection of wood panel images obtained using the visual inspection system, the first general wood panels semantic segmentation dataset is constructed for training the BiSeNetV1 model employed in this study. Furthermore, the calculation methods and processes for the essential key data required in the bark removal process are presented in detail. Comparative experiments of the BiSeNetV1 model and tests of bark removal effectiveness are conducted in both laboratory and sawmill environments. The results of the comparative experiments indicate that the application of the BiSeNetV1 segmentation model is rational and feasible. The results of the bark removal effectiveness tests demonstrate a significant improvement in both the quality and efficiency of bark removal. The developed equipment fully meets the sawmill's requirements for precision and efficiency in bark removal processing.

Title: A Prompt-Guided Spatio-Temporal Transformer Model for National-Wide Nuclear Radiation Forecasting

• Authors: Tengfei Lyu, Jindong Han, Hao Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11924
• Pdf URL: https://arxiv.org/pdf/2410.11924
• Copy Paste: [[2410.11924]] A Prompt-Guided Spatio-Temporal Transformer Model for National-Wide Nuclear Radiation Forecasting(https://arxiv.org/abs/2410.11924)
• Keywords: transformer
• Abstract: Nuclear radiation (NR), which refers to the energy emitted from atomic nuclei during decay, poses substantial risks to human health and environmental safety. Accurate forecasting of nuclear radiation levels is crucial for informed decision-making by both individuals and governments. However, this task is challenging due to the imbalanced distribution of monitoring stations over a wide spatial range and the non-stationary radiation variation patterns. In this study, we introduce NRFormer, an innovative framework tailored for national-wide prediction of nuclear radiation variations. By integrating a non-stationary temporal attention module, an imbalance-aware spatial attention module, and a radiation propagation prompting module, NRFormer collectively captures complex spatio-temporal dynamics of nuclear radiation. Extensive experiments on two real-world datasets demonstrate the superiority of our proposed framework against seven baselines. This research not only enhances the accuracy and reliability in nuclear radiation forecasting but also contributes to advancing emergency response strategies and monitoring systems, thereby safeguarding environmental and public health.

Title: Dual-frame Fluid Motion Estimation with Test-time Optimization and Zero-divergence Loss

• Authors: Yifei Zhang, Huan-ang Gao, Zhou Jiang, Hao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11934
• Pdf URL: https://arxiv.org/pdf/2410.11934
• Copy Paste: [[2410.11934]] Dual-frame Fluid Motion Estimation with Test-time Optimization and Zero-divergence Loss(https://arxiv.org/abs/2410.11934)
• Keywords: robust
• Abstract: 3D particle tracking velocimetry (PTV) is a key technique for analyzing turbulent flow, one of the most challenging computational problems of our century. At the core of 3D PTV is the dual-frame fluid motion estimation algorithm, which tracks particles across two consecutive frames. Recently, deep learning-based methods have achieved impressive accuracy in dual-frame fluid motion estimation; however, they heavily depend on large volumes of labeled data. In this paper, we introduce a new method that is completely self-supervised and notably outperforms its fully-supervised counterparts while requiring only 1% of the training samples (without labels) used by previous methods. Our method features a novel zero-divergence loss that is specific to the domain of turbulent flow. Inspired by the success of splat operation in high-dimensional filtering and random fields, we propose a splat-based implementation for this loss which is both efficient and effective. The self-supervised nature of our method naturally supports test-time optimization, leading to the development of a tailored Dynamic Velocimetry Enhancer (DVE) module. We demonstrate that strong cross-domain robustness is achieved through test-time optimization on unseen leave-one-out synthetic domains and real physical/biological domains. Code, data and models are available at this https URL.

Title: CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning

• Authors: Qingqing Cao, Mahyar Najibi, Sachin Mehta
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.11963
• Pdf URL: https://arxiv.org/pdf/2410.11963
• Copy Paste: [[2410.11963]] CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning(https://arxiv.org/abs/2410.11963)
• Keywords: robust, diffusion, large language model
• Abstract: Pretraining robust vision or multimodal foundation models (e.g., CLIP) relies on large-scale datasets that may be noisy, potentially misaligned, and have long-tail distributions. Previous works have shown promising results in augmenting datasets by generating synthetic samples. However, they only support domain-specific ad hoc use cases (e.g., either image or text only, but not both), and are limited in data diversity due to a lack of fine-grained control over the synthesis process. In this paper, we design a \emph{controllable} image-text synthesis pipeline, CtrlSynth, for data-efficient and robust multimodal learning. The key idea is to decompose the visual semantics of an image into basic elements, apply user-specified control policies (e.g., remove, add, or replace operations), and recompose them to synthesize images or texts. The decompose and recompose feature in CtrlSynth allows users to control data synthesis in a fine-grained manner by defining customized control policies to manipulate the basic elements. CtrlSynth leverages the capabilities of pretrained foundation models such as large language models or diffusion models to reason and recompose basic elements such that synthetic samples are natural and composed in diverse ways. CtrlSynth is a closed-loop, training-free, and modular framework, making it easy to support different pretrained models. With extensive experiments on 31 datasets spanning different vision and vision-language tasks, we show that CtrlSynth substantially improves zero-shot classification, image-text retrieval, and compositional reasoning performance of CLIP models.

Title: A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection

• Authors: James Enouen, Mahito Sugiyama
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.11964
• Pdf URL: https://arxiv.org/pdf/2410.11964
• Copy Paste: [[2410.11964]] A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection(https://arxiv.org/abs/2410.11964)
• Keywords: generative
• Abstract: The log-linear model has received a significant amount of theoretical attention in previous decades and remains the fundamental tool used for learning probability distributions over discrete variables. Despite its large popularity in statistical mechanics and high-dimensional statistics, the vast majority of such energy-based modeling approaches only focus on the two-variable relationships, such as Boltzmann machines and Markov graphical models. Although these approaches have easier-to-solve structure learning problems and easier-to-optimize parametric distributions, they often ignore the rich structure which exists in the higher-order interactions between different variables. Using more recent tools from the field of information geometry, we revisit the classical formulation of the log-linear model with a focus on higher-order mode interactions, going beyond the 1-body modes of independent distributions and the 2-body modes of Boltzmann distributions. This perspective allows us to define a complete decomposition of the KL error. This then motivates the formulation of a sparse selection problem over the set of possible mode interactions. In the same way as sparse graph selection allows for better generalization, we find that our learned distributions are able to more efficiently use the finite amount of data which is available in practice. On both synthetic and real-world datasets, we demonstrate our algorithm's effectiveness in maximizing the log-likelihood for the generative task and also the ease of adaptability to the discriminative task of classification.

Title: DDIL: Improved Diffusion Distillation With Imitation Learning

• Authors: Risheek Garrepalli, Shweta Mahajan, Munawar Hayat, Fatih Porikli
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11971
• Pdf URL: https://arxiv.org/pdf/2410.11971
• Copy Paste: [[2410.11971]] DDIL: Improved Diffusion Distillation With Imitation Learning(https://arxiv.org/abs/2410.11971)
• Keywords: diffusion, generative
• Abstract: Diffusion models excel at generative modeling (e.g., text-to-image) but sampling requires multiple denoising network passes, limiting practicality. Efforts such as progressive distillation or consistency distillation have shown promise by reducing the number of passes at the expense of quality of the generated samples. In this work we identify co-variate shift as one of reason for poor performance of multi-step distilled models from compounding error at inference time. To address co-variate shift, we formulate diffusion distillation within imitation learning (DDIL) framework and enhance training distribution for distilling diffusion models on both data distribution (forward diffusion) and student induced distributions (backward diffusion). Training on data distribution helps to diversify the generations by preserving marginal data distribution and training on student distribution addresses compounding error by correcting covariate shift. In addition, we adopt reflected diffusion formulation for distillation and demonstrate improved performance, stable training across different distillation methods. We show that DDIL consistency improves on baseline algorithms of progressive distillation (PD), Latent consistency models (LCM) and Distribution Matching Distillation (DMD2).

Title: The Fair Language Model Paradox

• Authors: Andrea Pinto, Tomer Galanti, Randall Balestriero
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11985
• Pdf URL: https://arxiv.org/pdf/2410.11985
• Copy Paste: [[2410.11985]] The Fair Language Model Paradox(https://arxiv.org/abs/2410.11985)
• Keywords: fair, large language model
• Abstract: Large Language Models (LLMs) are widely deployed in real-world applications, yet little is known about their training dynamics at the token level. Evaluation typically relies on aggregated training loss, measured at the batch level, which overlooks subtle per-token biases arising from (i) varying token-level dynamics and (ii) structural biases introduced by hyperparameters. While weight decay is commonly used to stabilize training, we reveal that it silently introduces performance biases detectable only at the token level. In fact, we empirically show across different dataset sizes, model architectures and sizes ranging from 270M to 3B parameters that as weight decay increases, low-frequency tokens are disproportionately depreciated. This is particularly concerning, as these neglected low-frequency tokens represent the vast majority of the token distribution in most languages, calling for novel regularization techniques that ensure fairness across all available tokens.

Title: Age-of-Gradient Updates for Federated Learning over Random Access Channels

• Authors: Yu Heng Wu, Houman Asgari, Stefano Rini, Andrea Munari
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.11986
• Pdf URL: https://arxiv.org/pdf/2410.11986
• Copy Paste: [[2410.11986]] Age-of-Gradient Updates for Federated Learning over Random Access Channels(https://arxiv.org/abs/2410.11986)
• Keywords: federate
• Abstract: This paper studies the problem of federated training of a deep neural network (DNN) over a random access channel (RACH) such as in computer networks, wireless networks, and cellular systems. More precisely, a set of remote users participate in training a centralized DNN model using SGD under the coordination of a parameter server (PS). The local model updates are transmitted from the remote users to the PS over a RACH using a slotted ALOHA protocol. The PS collects the updates from the remote users, accumulates them, and sends central model updates to the users at regular time intervals. We refer to this setting as the RACH-FL setting. The RACH-FL setting crucially addresses the problem of jointly designing a (i) client selection and (ii) gradient compression strategy which addresses the communication constraints between the remote users and the PS when transmission occurs over a RACH. For the RACH-FL setting, we propose a policy, which we term the ''age-of-gradient'' (AoG) policy in which (i) gradient sparsification is performed using top-K sparsification, (ii) the error correction is performed using memory accumulation, and (iii) the slot transmission probability is obtained by comparing the current local memory magnitude minus the magnitude of the gradient update to a threshold. Intuitively, the AoG measure of ''freshness'' of the memory state is reminiscent of the concept of age-of-information (AoI) in the context of communication theory and provides a rather natural interpretation of this policy. Numerical simulations show the superior performance of the AoG policy as compared to other RACH-FL policies.

Title: DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models

• Authors: Shangqian Gao, Chi-Heng Lin, Ting Hua, Tang Zheng, Yilin Shen, Hongxia Jin, Yen-Chang Hsu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11988
• Pdf URL: https://arxiv.org/pdf/2410.11988
• Copy Paste: [[2410.11988]] DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models(https://arxiv.org/abs/2410.11988)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, including language modeling, understanding, and generation. However, the increased memory and computational costs associated with these models pose significant challenges for deployment on resource-limited devices. Structural pruning has emerged as a promising solution to reduce the costs of LLMs without requiring post-processing steps. Prior structural pruning methods either follow the dependence of structures at the cost of limiting flexibility, or introduce non-trivial additional parameters by incorporating different projection matrices. In this work, we propose a novel approach that relaxes the constraint imposed by regular structural pruning methods and eliminates the structural dependence along the embedding dimension. Our dimension-independent structural pruning method offers several benefits. Firstly, our method enables different blocks to utilize different subsets of the feature maps. Secondly, by removing structural dependence, we facilitate each block to possess varying widths along its input and output dimensions, thereby significantly enhancing the flexibility of structural pruning. We evaluate our method on various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Experimental results demonstrate that our approach outperforms other state-of-the-art methods, showing for the first time that structural pruning can achieve an accuracy similar to semi-structural pruning.

Title: Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal Concepts

• Authors: Shaun Ho
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12001
• Pdf URL: https://arxiv.org/pdf/2410.12001
• Copy Paste: [[2410.12001]] Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal Concepts(https://arxiv.org/abs/2410.12001)
• Keywords: large language model
• Abstract: This paper aims to offer AI & Law researchers and practitioners a more detailed understanding of whether and how continued pre-training and instruction fine-tuning (IFT) of large language models (LLMs) on legal corpora increases their utilization of human-defined legal concepts when developing global contextual representations of input sequences. We compared three models: Mistral 7B, SaulLM-7B-Base (Mistral 7B with continued pre-training on legal corpora), and SaulLM-7B-Instruct (with further IFT). This preliminary assessment examined 7 distinct text sequences from recent AI & Law literature, each containing a human-defined legal concept. We first compared the proportions of total attention the models allocated to subsets of tokens representing the legal concepts. We then visualized patterns of raw attention score alterations, evaluating whether legal training introduced novel attention patterns corresponding to structures of human legal knowledge. This inquiry revealed that (1) the impact of legal training was unevenly distributed across the various human-defined legal concepts, and (2) the contextual representations of legal knowledge learned during legal training did not coincide with structures of human-defined legal concepts. We conclude with suggestions for further investigation into the dynamics of legal LLM training.

Title: Bias Similarity Across Large Language Models

• Authors: Hyejun Jeong, Shiqing Ma, Amir Houmansadr
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12010
• Pdf URL: https://arxiv.org/pdf/2410.12010
• Copy Paste: [[2410.12010]] Bias Similarity Across Large Language Models(https://arxiv.org/abs/2410.12010)
• Keywords: security, privacy, generative, large language model
• Abstract: Bias in machine learning models has been a chronic problem, especially as these models influence decision-making in human society. In generative AI, such as Large Language Models, the impact of bias is even more profound compared to the classification models. LLMs produce realistic and human-like content that users may unconsciously trust, which could perpetuate harmful stereotypes to the uncontrolled public. It becomes particularly concerning when utilized in journalism or education. While prior studies have explored and quantified bias in individual AI models, no work has yet compared bias similarity across different LLMs. To fill this gap, we take a comprehensive look at ten open- and closed-source LLMs from four model families, assessing the extent of biases through output distribution. Using two datasets-one containing 4k questions and another with one million questions for each of the four bias dimensions -- we measure functional similarity to understand how biases manifest across models. Our findings reveal that 1) fine-tuning does not significantly alter output distributions, which would limit its ability to mitigate bias, 2) LLMs within the same family tree do not produce similar output distributions, implying that addressing bias in one model could have limited implications for others in the same family, and 3) there is a possible risk of training data information leakage, raising concerns about privacy and data security. Our analysis provides insight into LLM behavior and highlights potential risks in real-world deployment.

Title: Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models

• Authors: Kushal Tatariya, Vladimir Araujo, Thomas Bauwens, Miryam de Lhoneux
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12011
• Pdf URL: https://arxiv.org/pdf/2410.12011
• Copy Paste: [[2410.12011]] Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models(https://arxiv.org/abs/2410.12011)
• Keywords: robust, transformer
• Abstract: Pixel-based language models have emerged as a compelling alternative to subword-based language modelling, particularly because they can represent virtually any script. PIXEL, a canonical example of such a model, is a vision transformer that has been pre-trained on rendered text. While PIXEL has shown promising cross-script transfer abilities and robustness to orthographic perturbations, it falls short of outperforming monolingual subword counterparts like BERT in most other contexts. This discrepancy raises questions about the amount of linguistic knowledge learnt by these models and whether their performance in language tasks stems more from their visual capabilities than their linguistic ones. To explore this, we probe PIXEL using a variety of linguistic and visual tasks to assess its position on the vision-to-language spectrum. Our findings reveal a substantial gap between the model's visual and linguistic understanding. The lower layers of PIXEL predominantly capture superficial visual features, whereas the higher layers gradually learn more syntactic and semantic abstractions. Additionally, we examine variants of PIXEL trained with different text rendering strategies, discovering that introducing certain orthographic constraints at the input level can facilitate earlier learning of surface-level features. With this study, we hope to provide insights that aid the further development of pixel-based language models.

Title: MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router

• Authors: Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, An Xu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12013
• Pdf URL: https://arxiv.org/pdf/2410.12013
• Copy Paste: [[2410.12013]] MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router(https://arxiv.org/abs/2410.12013)
• Keywords: large language model
• Abstract: Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model performance. Motivated by the recent observation of emergent large magnitude features in Large Language Models (LLM) and MoE routing policy, we propose MoE-Pruner, a method that prunes weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron. Our pruning method is one-shot, requiring no retraining or weight updates. We evaluate our method on Mixtral-8x7B and Mixtral-8x22B across multiple language benchmarks. Experimental results show that our pruning method significantly outperforms state-of-the-art LLM pruning methods. Furthermore, our pruned MoE models can benefit from a pretrained teacher model through expert-wise knowledge distillation, improving performance post-pruning. Experimental results demonstrate that the Mixtral-8x7B model with 50% sparsity maintains 99% of the performance of the original model after the expert-wise knowledge distillation.

Title: On Classification with Large Language Models in Cultural Analytics

• Authors: David Bamman, Kent K. Chang, Li Lucy, Naitian Zhou
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.12029
• Pdf URL: https://arxiv.org/pdf/2410.12029
• Copy Paste: [[2410.12029]] On Classification with Large Language Models in Cultural Analytics(https://arxiv.org/abs/2410.12029)
• Keywords: large language model
• Abstract: In this work, we survey the way in which classification is used as a sensemaking practice in cultural analytics, and assess where large language models can fit into this landscape. We identify ten tasks supported by publicly available datasets on which we empirically assess the performance of LLMs compared to traditional supervised methods, and explore the ways in which LLMs can be employed for sensemaking goals beyond mere accuracy. We find that prompt-based LLMs are competitive with traditional supervised models for established tasks, but perform less well on de novo tasks. In addition, LLMs can assist sensemaking by acting as an intermediary input to formal theory testing.

Title: A Survey on Deep Tabular Learning

• Authors: Shriyank Somvanshi, Subasish Das, Syed Aaqib Javed, Gian Antariksa, Ahmed Hossain
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12034
• Pdf URL: https://arxiv.org/pdf/2410.12034
• Copy Paste: [[2410.12034]] A Survey on Deep Tabular Learning(https://arxiv.org/abs/2410.12034)
• Keywords: robust, interpretability, diffusion, transformer
• Abstract: Tabular data, widely used in industries like healthcare, finance, and transportation, presents unique challenges for deep learning due to its heterogeneous nature and lack of spatial structure. This survey reviews the evolution of deep learning models for tabular data, from early fully connected networks (FCNs) to advanced architectures like TabNet, SAINT, TabTranSELU, and MambaNet. These models incorporate attention mechanisms, feature embeddings, and hybrid architectures to address tabular data complexities. TabNet uses sequential attention for instance-wise feature selection, improving interpretability, while SAINT combines self-attention and intersample attention to capture complex interactions across features and data points, both advancing scalability and reducing computational overhead. Hybrid architectures such as TabTransformer and FT-Transformer integrate attention mechanisms with multi-layer perceptrons (MLPs) to handle categorical and numerical data, with FT-Transformer adapting transformers for tabular datasets. Research continues to balance performance and efficiency for large datasets. Graph-based models like GNN4TDL and GANDALF combine neural networks with decision trees or graph structures, enhancing feature representation and mitigating overfitting in small datasets through advanced regularization techniques. Diffusion-based models like the Tabular Denoising Diffusion Probabilistic Model (TabDDPM) generate synthetic data to address data scarcity, improving model robustness. Similarly, models like TabPFN and Ptab leverage pre-trained language models, incorporating transfer learning and self-supervised techniques into tabular tasks. This survey highlights key advancements and outlines future research directions on scalability, generalization, and interpretability in diverse tabular data applications.

Title: Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstraction

• Authors: Kaiqiao Han, Tianqing Fang, Zhaowei Wang, Yangqiu Song, Mark Steedman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12040
• Pdf URL: https://arxiv.org/pdf/2410.12040
• Copy Paste: [[2410.12040]] Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstraction(https://arxiv.org/abs/2410.12040)
• Keywords: robust, large language model
• Abstract: While Large Language Models (LLMs) have showcased remarkable proficiency in reasoning, there is still a concern about hallucinations and unreliable reasoning issues due to semantic associations and superficial logical chains. To evaluate the extent to which LLMs perform robust reasoning instead of relying on superficial logical chains, we propose a new evaluation dataset, the Concept-Reversed Winograd Schema Challenge (CR-WSC), based on the famous Winograd Schema Challenge (WSC) dataset. By simply reversing the concepts to those that are more associated with the wrong answer, we find that the performance of LLMs drops significantly despite the rationale of reasoning remaining the same. Furthermore, we propose Abstraction-of-Thought (AoT), a novel prompt method for recovering adversarial cases to normal cases using conceptual abstraction to improve LLMs' robustness and consistency in reasoning, as demonstrated by experiments on CR-WSC.

Title: Differential Privacy on Trust Graphs

• Authors: Badih Ghazi, Ravi Kumar, Pasin Manurangsi, Serena Wang
• Subjects: cs.CR, cs.DS, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2410.12045
• Pdf URL: https://arxiv.org/pdf/2410.12045
• Copy Paste: [[2410.12045]] Differential Privacy on Trust Graphs(https://arxiv.org/abs/2410.12045)
• Keywords: privacy, robust
• Abstract: We study differential privacy (DP) in a multi-party setting where each party only trusts a (known) subset of the other parties with its data. Specifically, given a trust graph where vertices correspond to parties and neighbors are mutually trusting, we give a DP algorithm for aggregation with a much better privacy-utility trade-off than in the well-studied local model of DP (where each party trusts no other party). We further study a robust variant where each party trusts all but an unknown subset of at most $t$ of its neighbors (where $t$ is a given parameter), and give an algorithm for this setting. We complement our algorithms with lower bounds, and discuss implications of our work to other tasks in private learning and analytics.

Title: Skill-LLM: Repurposing General-Purpose LLMs for Skill Extraction

• Authors: Amirhossein Herandi, Yitao Li, Zhanlin Liu, Ximin Hu, Xiao Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12052
• Pdf URL: https://arxiv.org/pdf/2410.12052
• Copy Paste: [[2410.12052]] Skill-LLM: Repurposing General-Purpose LLMs for Skill Extraction(https://arxiv.org/abs/2410.12052)
• Keywords: extraction, large language model
• Abstract: Accurate skill extraction from job descriptions is crucial in the hiring process but remains challenging. Named Entity Recognition (NER) is a common approach used to address this issue. With the demonstrated success of large language models (LLMs) in various NLP tasks, including NER, we propose fine-tuning a specialized Skill-LLM and a light weight model to improve the precision and quality of skill extraction. In our study, we evaluated the fine-tuned Skill-LLM and the light weight model using a benchmark dataset and compared its performance against state-of-the-art (SOTA) methods. Our results show that this approach outperforms existing SOTA techniques.

Title: SOE: SO(3)-Equivariant 3D MRI Encoding

• Authors: Shizhe He, Magdalini Paschali, Jiahong Ouyang, Adnan Masood, Akshay Chaudhari, Ehsan Adeli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12053
• Pdf URL: https://arxiv.org/pdf/2410.12053
• Copy Paste: [[2410.12053]] SOE: SO(3)-Equivariant 3D MRI Encoding(https://arxiv.org/abs/2410.12053)
• Keywords: robust
• Abstract: Representation learning has become increasingly important, especially as powerful models have shifted towards learning latent representations before fine-tuning for downstream tasks. This approach is particularly valuable in leveraging the structural information within brain anatomy. However, a common limitation of recent models developed for MRIs is their tendency to ignore or remove geometric information, such as translation and rotation, thereby creating invariance with respect to geometric operations. We contend that incorporating knowledge about these geometric transformations into the model can significantly enhance its ability to learn more detailed anatomical information within brain structures. As a result, we propose a novel method for encoding 3D MRIs that enforces equivariance with respect to all rotations in 3D space, in other words, SO(3)-equivariance (SOE). By explicitly modeling this geometric equivariance in the representation space, we ensure that any rotational operation applied to the input image space is also reflected in the embedding representation space. This approach requires moving beyond traditional representation learning methods, as we need a representation vector space that allows for the application of the same SO(3) operation in that space. To facilitate this, we leverage the concept of vector neurons. The representation space formed by our method captures the brain's structural and anatomical information more effectively. We evaluate SOE pretrained on the structural MRIs of two public data sets with respect to the downstream task of predicting age and diagnosing Alzheimer's Disease from T1-weighted brain scans of the ADNI data set. We demonstrate that our approach not only outperforms other methods but is also robust against various degrees of rotation along different axes. The code is available at this https URL.

Title: Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned

• Authors: Cassandra L. Jacobs, Loïc Grobol, Alvin Tsang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12057
• Pdf URL: https://arxiv.org/pdf/2410.12057
• Copy Paste: [[2410.12057]] Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned(https://arxiv.org/abs/2410.12057)
• Keywords: generative
• Abstract: In this work we compare the generative behavior at the next token prediction level in several language models by comparing them to human productions in the cloze task. We find that while large models trained for longer are typically better estimators of human productions, but they reliably under-estimate the probabilities of human responses, over-rank rare responses, under-rank top responses, and produce highly distinct semantic spaces. Altogether, this work demonstrates in a tractable, interpretable domain that LM generations can not be used as replacements of or models of the cloze task.

Title: LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text

• Authors: Ben Hagag, Liav Harpaz, Gil Semo, Dor Bernsohn, Rohit Saha, Pashootan Vaezipoor, Kyryl Truskovskyi, Gerasimos Spanakis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12064
• Pdf URL: https://arxiv.org/pdf/2410.12064
• Copy Paste: [[2410.12064]] LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text(https://arxiv.org/abs/2410.12064)
• Keywords: privacy, protect
• Abstract: This paper presents the results of the LegalLens Shared Task, focusing on detecting legal violations within text in the wild across two sub-tasks: LegalLens-NER for identifying legal violation entities and LegalLens-NLI for associating these violations with relevant legal contexts and affected individuals. Using an enhanced LegalLens dataset covering labor, privacy, and consumer protection domains, 38 teams participated in the task. Our analysis reveals that while a mix of approaches was used, the top-performing teams in both tasks consistently relied on fine-tuning pre-trained language models, outperforming legal-specific models and few-shot methods. The top-performing team achieved a 7.11% improvement in NER over the baseline, while NLI saw a more marginal improvement of 5.7%. Despite these gains, the complexity of legal texts leaves room for further advancements.

Title: De-jargonizing Science for Journalists with GPT-4: A Pilot Study

• Authors: Sachita Nishal, Eric Lee, Nicholas Diakopoulos
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.12069
• Pdf URL: https://arxiv.org/pdf/2410.12069
• Copy Paste: [[2410.12069]] De-jargonizing Science for Journalists with GPT-4: A Pilot Study(https://arxiv.org/abs/2410.12069)
• Keywords: fair, generative, large language model
• Abstract: This study offers an initial evaluation of a human-in-the-loop system leveraging GPT-4 (a large language model or LLM), and Retrieval-Augmented Generation (RAG) to identify and define jargon terms in scientific abstracts, based on readers' self-reported knowledge. The system achieves fairly high recall in identifying jargon and preserves relative differences in readers' jargon identification, suggesting personalization as a feasible use-case for LLMs to support sense-making of complex information. Surprisingly, using only abstracts for context to generate definitions yields slightly more accurate and higher quality definitions than using RAG-based context from the fulltext of an article. The findings highlight the potential of generative AI for assisting science reporters, and can inform future work on developing tools to simplify dense documents.

Title: WeatherDG: LLM-assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation

• Authors: Chenghao Qian, Yuhu Guo, Yuhong Mo, Wenjing Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12075
• Pdf URL: https://arxiv.org/pdf/2410.12075
• Copy Paste: [[2410.12075]] WeatherDG: LLM-assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation(https://arxiv.org/abs/2410.12075)
• Keywords: diffusion, large language model, segmentation
• Abstract: In this work, we propose a novel approach, namely WeatherDG, that can generate realistic, weather-diverse, and driving-screen images based on the cooperation of two foundation models, i.e, Stable Diffusion (SD) and Large Language Model (LLM). Specifically, we first fine-tune the SD with source data, aligning the content and layout of generated samples with real-world driving scenarios. Then, we propose a procedural prompt generation method based on LLM, which can enrich scenario descriptions and help SD automatically generate more diverse, detailed images. In addition, we introduce a balanced generation strategy, which encourages the SD to generate high-quality objects of tailed classes under various weather conditions, such as riders and motorcycles. This segmentation-model-agnostic method can improve the generalization ability of existing models by additionally adapting them with the generated synthetic data. Experiments on three challenging datasets show that our method can significantly improve the segmentation performance of different state-of-the-art models on target domains. Notably, in the setting of ''Cityscapes to ACDC'', our method improves the baseline HRDA by 13.9% in mIoU.

Title: Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attacks

• Authors: Kevin Eykholt, Farhan Ahmed, Pratik Vaishnavi, Amir Rahmati
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.12076
• Pdf URL: https://arxiv.org/pdf/2410.12076
• Copy Paste: [[2410.12076]] Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attacks(https://arxiv.org/abs/2410.12076)
• Keywords: security, defense, attack, generative, large language model
• Abstract: The vulnerability of machine learning models in adversarial scenarios has garnered significant interest in the academic community over the past decade, resulting in a myriad of attacks and defenses. However, while the community appears to be overtly successful in devising new attacks across new contexts, the development of defenses has stalled. After a decade of research, we appear no closer to securing AI applications beyond additional training. Despite a lack of effective mitigations, AI development and its incorporation into existing systems charge full speed ahead with the rise of generative AI and large language models. Will our ineffectiveness in developing solutions to adversarial threats further extend to these new technologies? In this paper, we argue that overly permissive attack and overly restrictive defensive threat models have hampered defense development in the ML domain. Through the lens of adversarial evasion attacks against neural networks, we critically examine common attack assumptions, such as the ability to bypass any defense not explicitly built into the model. We argue that these flawed assumptions, seen as reasonable by the community based on paper acceptance, have encouraged the development of adversarial attacks that map poorly to real-world scenarios. In turn, new defenses evaluated against these very attacks are inadvertently required to be almost perfect and incorporated as part of the model. But do they need to? In practice, machine learning models are deployed as a small component of a larger system. We analyze adversarial machine learning from a system security perspective rather than an AI perspective and its implications for emerging AI paradigms.

Title: Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning

• Authors: Fengyu Gao, Ruida Zhou, Tianhao Wang, Cong Shen, Jing Yang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12085
• Pdf URL: https://arxiv.org/pdf/2410.12085
• Copy Paste: [[2410.12085]] Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning(https://arxiv.org/abs/2410.12085)
• Keywords: privacy, protect, large language model
• Abstract: Large Language Models (LLMs) rely on the contextual information embedded in examples/demonstrations to perform in-context learning (ICL). To mitigate the risk of LLMs potentially leaking private information contained in examples in the prompt, we introduce a novel data-adaptive differentially private algorithm called AdaDPSyn to generate synthetic examples from the private dataset and then use these synthetic examples to perform ICL. The objective of AdaDPSyn is to adaptively adjust the noise level in the data synthesis mechanism according to the inherent statistical properties of the data, thereby preserving high ICL accuracy while maintaining formal differential privacy guarantees. A key innovation in AdaDPSyn is the Precision-Focused Iterative Radius Reduction technique, which dynamically refines the aggregation radius - the scope of data grouping for noise addition - based on patterns observed in data clustering, thereby minimizing the amount of additive noise. We conduct extensive experiments on standard benchmarks and compare AdaDPSyn with DP few-shot generation algorithm (Tang et al., 2023). The experiments demonstrate that AdaDPSyn not only outperforms DP few-shot generation, but also maintains high accuracy levels close to those of non-private baselines, providing an effective solution for ICL with privacy protection.

Title: Bridging Large Language Models and Graph Structure Learning Models for Robust Representation Learning

• Authors: Guangxin Su, Yifan Zhu, Wenjie Zhang, Hanchen Wang, Ying Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12096
• Pdf URL: https://arxiv.org/pdf/2410.12096
• Copy Paste: [[2410.12096]] Bridging Large Language Models and Graph Structure Learning Models for Robust Representation Learning(https://arxiv.org/abs/2410.12096)
• Keywords: robust, large language model
• Abstract: Graph representation learning, involving both node features and graph structures, is crucial for real-world applications but often encounters pervasive noise. State-of-the-art methods typically address noise by focusing separately on node features with large language models (LLMs) and on graph structures with graph structure learning models (GSLMs). In this paper, we introduce LangGSL, a robust framework that integrates the complementary strengths of pre-trained language models and GSLMs to jointly enhance both node feature and graph structure learning. In LangGSL, we first leverage LLMs to filter noise in the raw data and extract valuable cleaned information as features, enhancing the synergy of downstream models. During the mutual learning phase in LangGSL, the core idea is to leverage the relatively small language model (LM) to process local attributes and generate reliable pseudo-labels and informative node embeddings, which are then integrated into the GSLM's prediction phase. This approach enriches the global context and enhances overall performance. Meanwhile, GSLM refines the evolving graph structure constructed from the LM's output, offering updated labels back to the LM as additional guidance, thus facilitating a more effective mutual learning process. The LM and GSLM work synergistically, complementing each other's strengths and offsetting weaknesses within a variational information-maximizing framework, resulting in enhanced node features and a more robust graph structure. Extensive experiments on diverse graph datasets of varying scales and across different task scenarios demonstrate the scalability and effectiveness of the proposed approach.

Title: The Persian Rug: solving toy models of superposition using large-scale symmetries

• Authors: Aditya Cowsik, Kfir Dolev, Alex Infanger
• Subjects: cs.LG, cond-mat.dis-nn, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12101
• Pdf URL: https://arxiv.org/pdf/2410.12101
• Copy Paste: [[2410.12101]] The Persian Rug: solving toy models of superposition using large-scale symmetries(https://arxiv.org/abs/2410.12101)
• Keywords: interpretability
• Abstract: We present a complete mechanistic description of the algorithm learned by a minimal non-linear sparse data autoencoder in the limit of large input dimension. The model, originally presented in arXiv:2209.10652, compresses sparse data vectors through a linear layer and decompresses using another linear layer followed by a ReLU activation. We notice that when the data is permutation symmetric (no input feature is privileged) large models reliably learn an algorithm that is sensitive to individual weights only through their large-scale statistics. For these models, the loss function becomes analytically tractable. Using this understanding, we give the explicit scalings of the loss at high sparsity, and show that the model is near-optimal among recently proposed architectures. In particular, changing or adding to the activation function any elementwise or filtering operation can at best improve the model's performance by a constant factor. Finally, we forward-engineer a model with the requisite symmetries and show that its loss precisely matches that of the trained models. Unlike the trained model weights, the low randomness in the artificial weights results in miraculous fractal structures resembling a Persian rug, to which the algorithm is oblivious. Our work contributes to neural network interpretability by introducing techniques for understanding the structure of autoencoders. Code to reproduce our results can be found at this https URL .

Title: OMCAT: Omni Context Aware Transformer

• Authors: Arushi Goel, Karan Sapra, Matthieu Le, Rafael Valle, Andrew Tao, Bryan Catanzaro
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12109
• Pdf URL: https://arxiv.org/pdf/2410.12109
• Copy Paste: [[2410.12109]] OMCAT: Omni Context Aware Transformer(https://arxiv.org/abs/2410.12109)
• Keywords: robust, transformer, large language model
• Abstract: Large Language Models (LLMs) have made significant strides in text generation and comprehension, with recent advancements extending into multimodal LLMs that integrate visual and audio inputs. However, these models continue to struggle with fine-grained, cross-modal temporal understanding, particularly when correlating events across audio and video streams. We address these challenges with two key contributions: a new dataset and model, called OCTAV and OMCAT respectively. OCTAV (Omni Context and Temporal Audio Video) is a novel dataset designed to capture event transitions across audio and video. Second, OMCAT (Omni Context Aware Transformer) is a powerful model that leverages RoTE (Rotary Time Embeddings), an innovative extension of RoPE, to enhance temporal grounding and computational efficiency in time-anchored tasks. Through a robust three-stage training pipeline-feature alignment, instruction tuning, and OCTAV-specific training-OMCAT excels in cross-modal temporal understanding. Our model demonstrates state-of-the-art performance on Audio-Visual Question Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in temporal reasoning and cross-modal alignment, as validated through comprehensive experiments and ablation studies. Our dataset and code will be made publicly available. The link to our demo page is this https URL.

Title: Scaling laws for post-training quantized large language models

• Authors: Zifei Xu, Alexander Lan, Wanzin Yazar, Tristan Webb, Sayeh Sharify, Xin Wang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12119
• Pdf URL: https://arxiv.org/pdf/2410.12119
• Copy Paste: [[2410.12119]] Scaling laws for post-training quantized large language models(https://arxiv.org/abs/2410.12119)
• Keywords: large language model
• Abstract: Generalization abilities of well-trained large language models (LLMs) are known to scale predictably as a function of model size. In contrast to the existence of practical scaling laws governing pre-training, the quality of LLMs after post-training compression remains highly unpredictable, often requiring case-by-case validation in practice. In this work, we attempted to close this gap for post-training weight quantization of LLMs by conducting a systematic empirical study on multiple LLM families quantized to numerous low-precision tensor data types using popular weight quantization techniques. We identified key scaling factors pertaining to characteristics of the local loss landscape, based on which the performance of quantized LLMs can be reasonably well predicted by a statistical model.

Title: Iter-AHMCL: Alleviate Hallucination for Large Language Model via Iterative Model-level Contrastive Learning

• Authors: Huiwen Wu, Xiaohan Li, Xiaogang Xu, Jiafei Wu, Deyi Zhang, Zhe Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12130
• Pdf URL: https://arxiv.org/pdf/2410.12130
• Copy Paste: [[2410.12130]] Iter-AHMCL: Alleviate Hallucination for Large Language Model via Iterative Model-level Contrastive Learning(https://arxiv.org/abs/2410.12130)
• Keywords: security, large language model
• Abstract: The development of Large Language Models (LLMs) has significantly advanced various AI applications in commercial and scientific research fields, such as scientific literature summarization, writing assistance, and knowledge graph construction. However, a significant challenge is the high risk of hallucination during LLM inference, which can lead to security concerns like factual inaccuracies, inconsistent information, and fabricated content. To tackle this issue, it is essential to develop effective methods for reducing hallucination while maintaining the original capabilities of the LLM. This paper introduces a novel approach called Iterative Model-level Contrastive Learning (Iter-AHMCL) to address hallucination. This method modifies the representation layers of pre-trained LLMs by using contrastive `positive' and `negative' models, trained on data with and without hallucinations. By leveraging the differences between these two models, we create a more straightforward pathway to eliminate hallucinations, and the iterative nature of contrastive learning further enhances performance. Experimental validation on four pre-trained foundation LLMs (LLaMA2, Alpaca, LLaMA3, and Qwen) finetuning with a specially designed dataset shows that our approach achieves an average improvement of 10.1 points on the TruthfulQA benchmark. Comprehensive experiments demonstrate the effectiveness of Iter-AHMCL in reducing hallucination while maintaining the general capabilities of LLMs.

Title: Preference Optimization with Multi-Sample Comparisons

• Authors: Chaoqi Wang, Zhuokai Zhao, Chen Zhu, Karthik Abinav Sankararaman, Michal Valko, Xuefei Cao, Zhaorun Chen, Madian Khabsa, Yuxin Chen, Hao Ma, Sinong Wang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12138
• Pdf URL: https://arxiv.org/pdf/2410.12138
• Copy Paste: [[2410.12138]] Preference Optimization with Multi-Sample Comparisons(https://arxiv.org/abs/2410.12138)
• Keywords: robust, diffusion, generative, large language model
• Abstract: Recent advancements in generative models, particularly large language models (LLMs) and diffusion models, have been driven by extensive pretraining on large datasets followed by post-training. However, current post-training methods such as reinforcement learning from human feedback (RLHF) and direct alignment from preference methods (DAP) primarily utilize single-sample comparisons. These approaches often fail to capture critical characteristics such as generative diversity and bias, which are more accurately assessed through multiple samples. To address these limitations, we introduce a novel approach that extends post-training to include multi-sample comparisons. To achieve this, we propose Multi-sample Direct Preference Optimization (mDPO) and Multi-sample Identity Preference Optimization (mIPO). These methods improve traditional DAP methods by focusing on group-wise characteristics. Empirically, we demonstrate that multi-sample comparison is more effective in optimizing collective characteristics~(e.g., diversity and bias) for generative models than single-sample comparison. Additionally, our findings suggest that multi-sample comparisons provide a more robust optimization framework, particularly for dataset with label noise.

Title: Layer-of-Thoughts Prompting (LoT): Leveraging LLM-Based Retrieval with Constraint Hierarchies

• Authors: Wachara Fungwacharakorn, Nguyen Ha Thanh, May Myo Zin, Ken Satoh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12153
• Pdf URL: https://arxiv.org/pdf/2410.12153
• Copy Paste: [[2410.12153]] Layer-of-Thoughts Prompting (LoT): Leveraging LLM-Based Retrieval with Constraint Hierarchies(https://arxiv.org/abs/2410.12153)
• Keywords: explainability, large language model
• Abstract: This paper presents a novel approach termed Layer-of-Thoughts Prompting (LoT), which utilizes constraint hierarchies to filter and refine candidate responses to a given query. By integrating these constraints, our method enables a structured retrieval process that enhances explainability and automation. Existing methods have explored various prompting techniques but often present overly generalized frameworks without delving into the nuances of prompts in multi-turn interactions. Our work addresses this gap by focusing on the hierarchical relationships among prompts. We demonstrate that the efficacy of thought hierarchy plays a critical role in developing efficient and interpretable retrieval algorithms. Leveraging Large Language Models (LLMs), LoT significantly improves the accuracy and comprehensibility of information retrieval tasks.

Title: Exploiting LLMs' Reasoning Capability to Infer Implicit Concepts in Legal Information Retrieval

• Authors: Hai-Long Nguyen, Tan-Minh Nguyen, Duc-Minh Nguyen, Thi-Hai-Yen Vuong, Ha-Thanh Nguyen, Xuan-Hieu Phan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12154
• Pdf URL: https://arxiv.org/pdf/2410.12154
• Copy Paste: [[2410.12154]] Exploiting LLMs' Reasoning Capability to Infer Implicit Concepts in Legal Information Retrieval(https://arxiv.org/abs/2410.12154)
• Keywords: large language model
• Abstract: Statutory law retrieval is a typical problem in legal language processing, that has various practical applications in law engineering. Modern deep learning-based retrieval methods have achieved significant results for this problem. However, retrieval systems relying on semantic and lexical correlations often exhibit limitations, particularly when handling queries that involve real-life scenarios, or use the vocabulary that is not specific to the legal domain. In this work, we focus on overcoming this weaknesses by utilizing the logical reasoning capabilities of large language models (LLMs) to identify relevant legal terms and facts related to the situation mentioned in the query. The proposed retrieval system integrates additional information from the term--based expansion and query reformulation to improve the retrieval accuracy. The experiments on COLIEE 2022 and COLIEE 2023 datasets show that extra knowledge from LLMs helps to improve the retrieval result of both lexical and semantic ranking models. The final ensemble retrieval system outperformed the highest results among all participating teams in the COLIEE 2022 and 2023 competitions.

Title: FragNet: A Graph Neural Network for Molecular Property Prediction with Four Layers of Interpretability

• Authors: Gihan Panapitiya, Peiyuan Gao, C Mark Maupin, Emily G Saldanha
• Subjects: cs.LG, cs.AI, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2410.12156
• Pdf URL: https://arxiv.org/pdf/2410.12156
• Copy Paste: [[2410.12156]] FragNet: A Graph Neural Network for Molecular Property Prediction with Four Layers of Interpretability(https://arxiv.org/abs/2410.12156)
• Keywords: interpretability
• Abstract: Molecular property prediction is a crucial step in many modern-day scientific applications including drug discovery and energy storage material design. Despite the availability of numerous machine learning models for this task, we are lacking in models that provide both high accuracies and interpretability of the predictions. We introduce the FragNet architecture, a graph neural network not only capable of achieving prediction accuracies comparable to the current state-of-the-art models, but also able to provide insight on four levels of molecular substructures. This model enables understanding of which atoms, bonds, molecular fragments, and molecular fragment connections are critical in the prediction of a given molecular property. The ability to interpret the importance of connections between fragments is of particular interest for molecules which have substructures that are not connected with regular covalent bonds. The interpretable capabilities of FragNet are key to gaining scientific insights from the model's learned patterns between molecular structure and molecular properties.

Title: SAM-Guided Masked Token Prediction for 3D Scene Understanding

• Authors: Zhimin Chen, Liang Yang, Yingwei Li, Longlong Jing, Bing Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12158
• Pdf URL: https://arxiv.org/pdf/2410.12158
• Copy Paste: [[2410.12158]] SAM-Guided Masked Token Prediction for 3D Scene Understanding(https://arxiv.org/abs/2410.12158)
• Keywords: transformer, segmentation
• Abstract: Foundation models have significantly enhanced 2D task performance, and recent works like Bridge3D have successfully applied these models to improve 3D scene understanding through knowledge distillation, marking considerable advancements. Nonetheless, challenges such as the misalignment between 2D and 3D representations and the persistent long-tail distribution in 3D datasets still restrict the effectiveness of knowledge distillation from 2D to 3D using foundation models. To tackle these issues, we introduce a novel SAM-guided tokenization method that seamlessly aligns 3D transformer structures with region-level knowledge distillation, replacing the traditional KNN-based tokenization techniques. Additionally, we implement a group-balanced re-weighting strategy to effectively address the long-tail problem in knowledge distillation. Furthermore, inspired by the recent success of masked feature prediction, our framework incorporates a two-stage masked token prediction process in which the student model predicts both the global embeddings and the token-wise local embeddings derived from the teacher models trained in the first stage. Our methodology has been validated across multiple datasets, including SUN RGB-D, ScanNet, and S3DIS, for tasks like 3D object detection and semantic segmentation. The results demonstrate significant improvements over current State-of-the-art self-supervised methods, establishing new benchmarks in this field.

Title: NSSI-Net: Multi-Concept Generative Adversarial Network for Non-Suicidal Self-Injury Detection Using High-Dimensional EEG Signals in a Semi-Supervised Learning Framework

• Authors: Zhen Liang, Weishan Ye, Qile Liu, Li Zhang, Gan Huang, Yongjie Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12159
• Pdf URL: https://arxiv.org/pdf/2410.12159
• Copy Paste: [[2410.12159]] NSSI-Net: Multi-Concept Generative Adversarial Network for Non-Suicidal Self-Injury Detection Using High-Dimensional EEG Signals in a Semi-Supervised Learning Framework(https://arxiv.org/abs/2410.12159)
• Keywords: extraction, generative
• Abstract: Non-suicidal self-injury (NSSI) is a serious threat to the physical and mental health of adolescents, significantly increasing the risk of suicide and attracting widespread public concern. Electroencephalography (EEG), as an objective tool for identifying brain disorders, holds great promise. However, extracting meaningful and reliable features from high-dimensional EEG data, especially by integrating spatiotemporal brain dynamics into informative representations, remains a major challenge. In this study, we introduce an advanced semi-supervised adversarial network, NSSI-Net, to effectively model EEG features related to NSSI. NSSI-Net consists of two key modules: a spatial-temporal feature extraction module and a multi-concept discriminator. In the spatial-temporal feature extraction module, an integrated 2D convolutional neural network (2D-CNN) and a bi-directional Gated Recurrent Unit (BiGRU) are used to capture both spatial and temporal dynamics in EEG data. In the multi-concept discriminator, signal, gender, domain, and disease levels are fully explored to extract meaningful EEG features, considering individual, demographic, disease variations across a diverse population. Based on self-collected NSSI data (n=114), the model's effectiveness and reliability are demonstrated, with a 7.44% improvement in performance compared to existing machine learning and deep learning methods. This study advances the understanding and early diagnosis of NSSI in adolescents with depression, enabling timely intervention. The source code is available at this https URL.

Title: Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Generator-Validator Fine-tuning

• Authors: Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Dongmei Zhang, Surajit Chaudhuri
• Subjects: cs.CL, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12164
• Pdf URL: https://arxiv.org/pdf/2410.12164
• Copy Paste: [[2410.12164]] Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Generator-Validator Fine-tuning(https://arxiv.org/abs/2410.12164)
• Keywords: generative
• Abstract: In this work, we propose Table-LLM-Specialist, or Table-Specialist for short, as a new self-trained fine-tuning paradigm specifically designed for table tasks. Our insight is that for each table task, there often exist two dual versions of the same task, one generative and one classification in nature. Leveraging their duality, we propose a Generator-Validator paradigm, to iteratively generate-then-validate training data from language-models, to fine-tune stronger \sys models that can specialize in a given task, without requiring manually-labeled data. Our extensive evaluations suggest that our Table-Specialist has (1) \textit{strong performance} on diverse table tasks over vanilla language-models -- for example, Table-Specialist fine-tuned on GPT-3.5 not only outperforms vanilla GPT-3.5, but can often match or surpass GPT-4 level quality, (2) \textit{lower cost} to deploy, because when Table-Specialist fine-tuned on GPT-3.5 achieve GPT-4 level quality, it becomes possible to deploy smaller models with lower latency and inference cost, with comparable quality, and (3) \textit{better generalizability} when evaluated across multiple benchmarks, since \sys is fine-tuned on a broad range of training data systematically generated from diverse real tables. Our code and data will be available at this https URL.

Title: Exploring Large Language Models for Hate Speech Detection in Rioplatense Spanish

• Authors: Juan Manuel Pérez, Paula Miguel, Viviana Cotik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12174
• Pdf URL: https://arxiv.org/pdf/2410.12174
• Copy Paste: [[2410.12174]] Exploring Large Language Models for Hate Speech Detection in Rioplatense Spanish(https://arxiv.org/abs/2410.12174)
• Keywords: large language model
• Abstract: Hate speech detection deals with many language variants, slang, slurs, expression modalities, and cultural nuances. This outlines the importance of working with specific corpora, when addressing hate speech within the scope of Natural Language Processing, recently revolutionized by the irruption of Large Language Models. This work presents a brief analysis of the performance of large language models in the detection of Hate Speech for Rioplatense Spanish. We performed classification experiments leveraging chain-of-thought reasoning with ChatGPT 3.5, Mixtral, and Aya, comparing their results with those of a state-of-the-art BERT classifier. These experiments outline that, even if large language models show a lower precision compared to the fine-tuned BERT classifier and, in some cases, they find hard-to-get slurs or colloquialisms, they still are sensitive to highly nuanced cases (particularly, homophobic/transphobic hate speech). We make our code and models publicly available for future research.

Title: Reinforcement Learning with LTL and $\omega$-Regular Objectives via Optimality-Preserving Translation to Average Rewards

• Authors: Xuan-Bach Le, Dominik Wagner, Leon Witzman, Alexander Rabinovich, Luke Ong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12175
• Pdf URL: https://arxiv.org/pdf/2410.12175
• Copy Paste: [[2410.12175]] Reinforcement Learning with LTL and $\omega$-Regular Objectives via Optimality-Preserving Translation to Average Rewards(https://arxiv.org/abs/2410.12175)
• Keywords: explainability
• Abstract: Linear temporal logic (LTL) and, more generally, $\omega$-regular objectives are alternatives to the traditional discount sum and average reward objectives in reinforcement learning (RL), offering the advantage of greater comprehensibility and hence explainability. In this work, we study the relationship between these objectives. Our main result is that each RL problem for $\omega$-regular objectives can be reduced to a limit-average reward problem in an optimality-preserving fashion, via (finite-memory) reward machines. Furthermore, we demonstrate the efficacy of this approach by showing that optimal policies for limit-average problems can be found asymptotically by solving a sequence of discount-sum problems approximately. Consequently, we resolve an open problem: optimal policies for LTL and $\omega$-regular objectives can be learned asymptotically.

Title: ExoTST: Exogenous-Aware Temporal Sequence Transformer for Time Series Prediction

• Authors: Kshitij Tayal, Arvind Renganathan, Xiaowei Jia, Vipin Kumar, Dan Lu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12184
• Pdf URL: https://arxiv.org/pdf/2410.12184
• Copy Paste: [[2410.12184]] ExoTST: Exogenous-Aware Temporal Sequence Transformer for Time Series Prediction(https://arxiv.org/abs/2410.12184)
• Keywords: robust, transformer
• Abstract: Accurate long-term predictions are the foundations for many machine learning applications and decision-making processes. Traditional time series approaches for prediction often focus on either autoregressive modeling, which relies solely on past observations of the target ``endogenous variables'', or forward modeling, which considers only current covariate drivers ``exogenous variables''. However, effectively integrating past endogenous and past exogenous with current exogenous variables remains a significant challenge. In this paper, we propose ExoTST, a novel transformer-based framework that effectively incorporates current exogenous variables alongside past context for improved time series prediction. To integrate exogenous information efficiently, ExoTST leverages the strengths of attention mechanisms and introduces a novel cross-temporal modality fusion module. This module enables the model to jointly learn from both past and current exogenous series, treating them as distinct modalities. By considering these series separately, ExoTST provides robustness and flexibility in handling data uncertainties that arise from the inherent distribution shift between historical and current exogenous variables. Extensive experiments on real-world carbon flux datasets and time series benchmarks demonstrate ExoTST's superior performance compared to state-of-the-art baselines, with improvements of up to 10\% in prediction accuracy. Moreover, ExoTST exhibits strong robustness against missing values and noise in exogenous drivers, maintaining consistent performance in real-world situations where these imperfections are common.

Title: DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs

• Authors: Yingsong Luo, Ling Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12187
• Pdf URL: https://arxiv.org/pdf/2410.12187
• Copy Paste: [[2410.12187]] DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs(https://arxiv.org/abs/2410.12187)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel in various tasks but face deployment challenges due to hardware constraints. We propose density-aware post-training weight-only quantization (DAQ), which has two stages: 1) density-centric alignment, which identifies the center of high-density weights and centers the dynamic range on this point to align high-density weight regions with floating-point high-precision regions; 2) learnable dynamic range adjustment, which adjusts the dynamic range by optimizing quantization parameters (i.e., scale and zero-point) based on the impact of weights on the model output. Experiments on LLaMA and LLaMA-2 show that DAQ consistently outperforms the best baseline method, reducing perplexity loss by an average of 22.8% on LLaMA and 19.6% on LLaMA-2. Our code is available at this https URL.

Title: LPUF-AuthNet: A Lightweight PUF-Based IoT Authentication via Tandem Neural Networks and Split Learning

• Authors: Brahim Mefgouda, Raviha Khan, Omar Alhussein, Hani Saleh, Hossien B. Eldeeb, Anshul Pandey, Sami Muhaidat
• Subjects: cs.CR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2410.12190
• Pdf URL: https://arxiv.org/pdf/2410.12190
• Copy Paste: [[2410.12190]] LPUF-AuthNet: A Lightweight PUF-Based IoT Authentication via Tandem Neural Networks and Split Learning(https://arxiv.org/abs/2410.12190)
• Keywords: secure, security, attack, robust
• Abstract: By 2025, the internet of things (IoT) is projected to connect over 75 billion devices globally, fundamentally altering how we interact with our environments in both urban and rural settings. However, IoT device security remains challenging, particularly in the authentication process. Traditional cryptographic methods often struggle with the constraints of IoT devices, such as limited computational power and storage. This paper considers physical unclonable functions (PUFs) as robust security solutions, utilizing their inherent physical uniqueness to authenticate devices securely. However, traditional PUF systems are vulnerable to machine learning (ML) attacks and burdened by large datasets. Our proposed solution introduces a lightweight PUF mechanism, called LPUF-AuthNet, combining tandem neural networks (TNN) with a split learning (SL) paradigm. The proposed approach provides scalability, supports mutual authentication, and enhances security by resisting various types of attacks, paving the way for secure integration into future 6G technologies.

Title: Negative-Prompt-driven Alignment for Generative Language Model

• Authors: Shiqi Qiao, Ning Xv, Biao Liu, Xin Geng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12194
• Pdf URL: https://arxiv.org/pdf/2410.12194
• Copy Paste: [[2410.12194]] Negative-Prompt-driven Alignment for Generative Language Model(https://arxiv.org/abs/2410.12194)
• Keywords: generative, large language model
• Abstract: Large language models have achieved remarkable capabilities, but aligning their outputs with human values and preferences remains a significant challenge. Existing alignment methods primarily focus on positive examples while overlooking the importance of negative responses in guiding models away from undesirable behaviors. For instance, the widely-used alignment datasets reveals a scarcity of explicit negative examples that contradict human values, hindering its ability to discourage harmful or biased outputs during training. To address this limitation, we propose NEAT, i.e., NEgative-prompt-driven AlignmenT, to introduce negative prompts to generate undesirable responses alongside positive examples during the optimization process. NEAT explicitly penalizes the model for producing harmful outputs, guiding it not only toward desirable behaviors but also steering it away from generating undesirable, biased responses. This dual feedback mechanism enables better alignment with human preferences, crucial in contexts where avoiding harm is paramount. Starting from a pre-trained language model, NEAT performs online alignment by incorporating a ranking loss derived from an expanded preference dataset containing both positive and negative examples. Extensive experiments validate NEAT's effectiveness in significantly enhancing language models' alignment with human values and preferences.

Title: Sparse Prototype Network for Explainable Pedestrian Behavior Prediction

• Authors: Yan Feng, Alexander Carballo, Kazuya Takeda
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12195
• Pdf URL: https://arxiv.org/pdf/2410.12195
• Copy Paste: [[2410.12195]] Sparse Prototype Network for Explainable Pedestrian Behavior Prediction(https://arxiv.org/abs/2410.12195)
• Keywords: explainability
• Abstract: Predicting pedestrian behavior is challenging yet crucial for applications such as autonomous driving and smart city. Recent deep learning models have achieved remarkable performance in making accurate predictions, but they fail to provide explanations of their inner workings. One reason for this problem is the multi-modal inputs. To bridge this gap, we present Sparse Prototype Network (SPN), an explainable method designed to simultaneously predict a pedestrian's future action, trajectory, and pose. SPN leverages an intermediate prototype bottleneck layer to provide sample-based explanations for its predictions. The prototypes are modality-independent, meaning that they can correspond to any modality from the input. Therefore, SPN can extend to arbitrary combinations of modalities. Regularized by mono-semanticity and clustering constraints, the prototypes learn consistent and human-understandable features and achieve state-of-the-art performance on action, trajectory and pose prediction on TITAN and PIE. Finally, we propose a metric named Top-K Mono-semanticity Scale to quantitatively evaluate the explainability. Qualitative results show the positive correlation between sparsity and explainability. Code available at this https URL.

Title: Abnormality Forecasting: Time Series Anomaly Prediction via Future Context Modeling

• Authors: Sinong Zhao, Wenrui Wang, Hongzuo Xu, Zhaoyang Yu, Qingsong Wen, Gang Wang, xiaoguang Liu, Guansong Pang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12206
• Pdf URL: https://arxiv.org/pdf/2410.12206
• Copy Paste: [[2410.12206]] Abnormality Forecasting: Time Series Anomaly Prediction via Future Context Modeling(https://arxiv.org/abs/2410.12206)
• Keywords: security
• Abstract: Identifying anomalies from time series data plays an important role in various fields such as infrastructure security, intelligent operation and maintenance, and space exploration. Current research focuses on detecting the anomalies after they occur, which can lead to significant financial/reputation loss or infrastructure damage. In this work we instead study a more practical yet very challenging problem, time series anomaly prediction, aiming at providing early warnings for abnormal events before their occurrence. To tackle this problem, we introduce a novel principled approach, namely future context modeling (FCM). Its key insight is that the future abnormal events in a target window can be accurately predicted if their preceding observation window exhibits any subtle difference to normal data. To effectively capture such differences, FCM first leverages long-term forecasting models to generate a discriminative future context based on the observation data, aiming to amplify those subtle but unusual difference. It then models a normality correlation of the observation data with the forecasting future context to complement the normality modeling of the observation data in foreseeing possible abnormality in the target window. A joint variate-time attention learning is also introduced in FCM to leverage both temporal signals and features of the time series data for more discriminative normality modeling in the aforementioned two views. Comprehensive experiments on five datasets demonstrate that FCM gains good recall rate (70\%+) on multiple datasets and significantly outperforms all baselines in F1 score. Code is available at this https URL.

Title: Order-Aware Interactive Segmentation

• Authors: Bin Wang, Anwesa Choudhuri, Meng Zheng, Zhongpai Gao, Benjamin Planche, Andong Deng, Qin Liu, Terrence Chen, Ulas Bagci, Ziyan Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12214
• Pdf URL: https://arxiv.org/pdf/2410.12214
• Copy Paste: [[2410.12214]] Order-Aware Interactive Segmentation(https://arxiv.org/abs/2410.12214)
• Keywords: segmentation
• Abstract: Interactive segmentation aims to accurately segment target objects with minimal user interactions. However, current methods often fail to accurately separate target objects from the background, due to a limited understanding of order, the relative depth between objects in a scene. To address this issue, we propose OIS: order-aware interactive segmentation, where we explicitly encode the relative depth between objects into order maps. We introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) to attend to the image features. We further present an object-aware attention module to incorporate a strong object-level understanding to better differentiate objects with similar order. Our approach allows both dense and sparse integration of user clicks, enhancing both accuracy and efficiency as compared to prior works. Experimental results demonstrate that OIS achieves state-of-the-art performance, improving mIoU after one click by 7.61 on the HQSeg44K dataset and 1.32 on the DAVIS dataset as compared to the previous state-of-the-art SegNext, while also doubling inference speed compared to current leading methods. The project page is this https URL

Title: On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation

• Authors: Xiaonan Jing, Srinivas Billa, Danny Godbout
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12222
• Pdf URL: https://arxiv.org/pdf/2410.12222
• Copy Paste: [[2410.12222]] On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation(https://arxiv.org/abs/2410.12222)
• Keywords: large language model
• Abstract: Hallucination has been a popular topic in natural language generation (NLG). In real-world applications, unfaithful content can result in bad data quality or loss of trust from end users. Thus, it is crucial to fact-check before adopting NLG for production usage, which can be expensive if done manually. In this paper, we investigate automated faithfulness evaluation in guided NLG. We developed a rubrics template and use large language models (LLMs) to score the generation into quantifiable scales. We compared popular LLMs as well as the widely adopted natural language inference (NLI) models in scoring quality and sensitivity. In addition, we developed methods to generation synthetic unfaithful data, as well as a heuristics to quantify the percentage of hallucination. Our results on 4 travel-domain industry dataset show that GPT-4 can provide accurate judgement and explanation on whether a source and a generation are factually consistent. Furthermore, we found that tuning NLI models on synthetic data can improve performance. Lastly, we present insights on latency and cost for deploying such system.

Title: Causally-Aware Unsupervised Feature Selection Learning

• Authors: Zongxin Shen, Yanyong Huang, Minbo Ma, Tianrui Li
• Subjects: cs.LG, stat.ME
• Abstract URL: https://arxiv.org/abs/2410.12224
• Pdf URL: https://arxiv.org/pdf/2410.12224
• Copy Paste: [[2410.12224]] Causally-Aware Unsupervised Feature Selection Learning(https://arxiv.org/abs/2410.12224)
• Keywords: interpretability
• Abstract: Unsupervised feature selection (UFS) has recently gained attention for its effectiveness in processing unlabeled high-dimensional data. However, existing methods overlook the intrinsic causal mechanisms within the data, resulting in the selection of irrelevant features and poor interpretability. Additionally, previous graph-based methods fail to account for the differing impacts of non-causal and causal features in constructing the similarity graph, which leads to false links in the generated graph. To address these issues, a novel UFS method, called Causally-Aware UnSupErvised Feature Selection learning (CAUSE-FS), is proposed. CAUSE-FS introduces a novel causal regularizer that reweights samples to balance the confounding distribution of each treatment feature. This regularizer is subsequently integrated into a generalized unsupervised spectral regression model to mitigate spurious associations between features and clustering labels, thus achieving causal feature selection. Furthermore, CAUSE-FS employs causality-guided hierarchical clustering to partition features with varying causal contributions into multiple granularities. By integrating similarity graphs learned adaptively at different granularities, CAUSE-FS increases the importance of causal features when constructing the fused similarity graph to capture the reliable local structure of data. Extensive experimental results demonstrate the superiority of CAUSE-FS over state-of-the-art methods, with its interpretability further validated through feature visualization.

Title: Enhancing LLM Agents for Code Generation with Possibility and Pass-rate Prioritized Experience Replay

• Authors: Yuyang Chen, Kaiyan Zhao, Yiming Wang, Ming Yang, Jian Zhang, Xiaoguang Niu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12236
• Pdf URL: https://arxiv.org/pdf/2410.12236
• Copy Paste: [[2410.12236]] Enhancing LLM Agents for Code Generation with Possibility and Pass-rate Prioritized Experience Replay(https://arxiv.org/abs/2410.12236)
• Keywords: transformer, large language model
• Abstract: Nowadays transformer-based Large Language Models (LLM) for code generation tasks usually apply sampling and filtering pipelines. Due to the sparse reward problem in code generation tasks caused by one-token incorrectness, transformer-based models will sample redundant programs till they find a correct one, leading to low efficiency. To overcome the challenge, we incorporate Experience Replay (ER) in the fine-tuning phase, where codes and programs produced are stored and will be replayed to give the LLM agent a chance to learn from past experiences. Based on the spirit of ER, we introduce a novel approach called BTP pipeline which consists of three phases: beam search sampling, testing phase, and prioritized experience replay phase. The approach makes use of failed programs collected by code models and replays programs with high Possibility and Pass-rate Prioritized value (P2Value) from the replay buffer to improve efficiency. P2Value comprehensively considers the possibility of transformers' output and pass rate and can make use of the redundant resources caused by the problem that most programs collected by LLMs fail to pass any tests. We empirically apply our approach in several LLMs, demonstrating that it enhances their performance in code generation tasks and surpasses existing baselines.

Title: Off-dynamics Conditional Diffusion Planners

• Authors: Wen Zheng Terence Ng, Jianda Chen, Tianwei Zhang
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.12238
• Pdf URL: https://arxiv.org/pdf/2410.12238
• Copy Paste: [[2410.12238]] Off-dynamics Conditional Diffusion Planners(https://arxiv.org/abs/2410.12238)
• Keywords: robust, diffusion
• Abstract: Offline Reinforcement Learning (RL) offers an attractive alternative to interactive data acquisition by leveraging pre-existing datasets. However, its effectiveness hinges on the quantity and quality of the data samples. This work explores the use of more readily available, albeit off-dynamics datasets, to address the challenge of data scarcity in Offline RL. We propose a novel approach using conditional Diffusion Probabilistic Models (DPMs) to learn the joint distribution of the large-scale off-dynamics dataset and the limited target dataset. To enable the model to capture the underlying dynamics structure, we introduce two contexts for the conditional model: (1) a continuous dynamics score allows for partial overlap between trajectories from both datasets, providing the model with richer information; (2) an inverse-dynamics context guides the model to generate trajectories that adhere to the target environment's dynamic constraints. Empirical results demonstrate that our method significantly outperforms several strong baselines. Ablation studies further reveal the critical role of each dynamics context. Additionally, our model demonstrates that by modifying the context, we can interpolate between source and target dynamics, making it more robust to subtle shifts in the environment.

Title: Leveraging Spatial Attention and Edge Context for Optimized Feature Selection in Visual Localization

• Authors: Nanda Febri Istighfarin, HyungGi Jo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12240
• Pdf URL: https://arxiv.org/pdf/2410.12240
• Copy Paste: [[2410.12240]] Leveraging Spatial Attention and Edge Context for Optimized Feature Selection in Visual Localization(https://arxiv.org/abs/2410.12240)
• Keywords: robust
• Abstract: Visual localization determines an agent's precise position and orientation within an environment using visual data. It has become a critical task in the field of robotics, particularly in applications such as autonomous navigation. This is due to the ability to determine an agent's pose using cost-effective sensors such as RGB cameras. Recent methods in visual localization employ scene coordinate regression to determine the agent's pose. However, these methods face challenges as they attempt to regress 2D-3D correspondences across the entire image region, despite not all regions providing useful information. To address this issue, we introduce an attention network that selectively targets informative regions of the image. Using this network, we identify the highest-scoring features to improve the feature selection process and combine the result with edge detection. This integration ensures that the features chosen for the training buffer are located within robust regions, thereby improving 2D-3D correspondence and overall localization performance. Our approach was tested on the outdoor benchmark dataset, demonstrating superior results compared to previous methods.

Title: EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference

• Authors: Yulei Qian, Fengcun Li, Xiangyang Ji, Xiaoyu Zhao, Jianchao Tan, Kefeng Zhang, Xunliang Cai
• Subjects: cs.CL, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.12247
• Pdf URL: https://arxiv.org/pdf/2410.12247
• Copy Paste: [[2410.12247]] EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference(https://arxiv.org/abs/2410.12247)
• Keywords: large language model
• Abstract: Large Language Model (LLM) has revolutionized the field of artificial intelligence, with their capabilities expanding rapidly due to advances in deep learning and increased computational resources. The mixture-of-experts (MoE) model has emerged as a prominent architecture in the field of LLM, better balancing the model performance and computational efficiency. MoE architecture allows for effective scaling and efficient parallel processing, but the GEMM (General Matrix Multiply) of MoE and the large parameters introduce challenges in terms of computation efficiency and communication overhead, which becomes the throughput bottleneck during inference. Applying a single parallelism strategy like EP, DP, PP, etc. to MoE architecture usually achieves sub-optimal inference throughput, the straightforward combinations of existing different parallelisms on MoE can not obtain optimal inference throughput yet. This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE that goes beyond the existing inference parallelism schemes. Our approach focuses on optimizing the computation of MoE FFN (FeedForward Network) modules by dynamically selecting the best kernel implementation of GroupGemm and DenseGemm for different loads and adaptively overlapping these computations with \textit{all2all} communication, leading to a substantial increase in throughput. Our experimental results demonstrate an average 21% improvement in prefill throughput over existing parallel inference methods. Specifically, we validated our method on DeepSeekV2, a highly optimized model claimed to achieve a prefill throughput of 100K tokens per second. By applying EPS-MoE, we further accelerated it to at least 120K tokens per second.

Title: CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity

• Authors: Jintao Liu, Ruixue Ding, Linhao Zhang, Pengjun Xie, Fie Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12248
• Pdf URL: https://arxiv.org/pdf/2410.12248
• Copy Paste: [[2410.12248]] CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity(https://arxiv.org/abs/2410.12248)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) aims to enhance large language models (LLMs) to generate more accurate and reliable answers with the help of the retrieved context from external knowledge sources, thereby reducing the incidence of hallucinations. Despite the advancements, evaluating these systems remains a crucial research area due to the following issues: (1) Limited data diversity: The insufficient diversity of knowledge sources and query types constrains the applicability of RAG systems; (2) Obscure problems location: Existing evaluation methods have difficulty in locating the stage of the RAG pipeline where problems occur; (3) Unstable retrieval evaluation: These methods often fail to effectively assess retrieval performance, particularly when the chunking strategy changes. To tackle these challenges, we propose a Comprehensive Full-chain Evaluation (CoFE-RAG) framework to facilitate thorough evaluation across the entire RAG pipeline, including chunking, retrieval, reranking, and generation. To effectively evaluate the first three phases, we introduce multi-granularity keywords, including coarse-grained and fine-grained keywords, to assess the retrieved context instead of relying on the annotation of golden chunks. Moreover, we release a holistic benchmark dataset tailored for diverse data scenarios covering a wide range of document formats and query types. We demonstrate the utility of the CoFE-RAG framework by conducting experiments to evaluate each stage of RAG systems. Our evaluation method provides unique insights into the effectiveness of RAG systems in handling diverse data scenarios, offering a more nuanced understanding of their capabilities and limitations.

Title: Dual Action Policy for Robust Sim-to-Real Reinforcement Learning

• Authors: Ng Wen Zheng Terence, Chen Jianda
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.12250
• Pdf URL: https://arxiv.org/pdf/2410.12250
• Copy Paste: [[2410.12250]] Dual Action Policy for Robust Sim-to-Real Reinforcement Learning(https://arxiv.org/abs/2410.12250)
• Keywords: robust
• Abstract: This paper presents Dual Action Policy (DAP), a novel approach to address the dynamics mismatch inherent in the sim-to-real gap of reinforcement learning. DAP uses a single policy to predict two sets of actions: one for maximizing task rewards in simulation and another specifically for domain adaptation via reward adjustments. This decoupling makes it easier to maximize the overall reward in the source domain during training. Additionally, DAP incorporates uncertainty-based exploration during training to enhance agent robustness. Experimental results demonstrate DAP's effectiveness in bridging the sim-to-real gap, outperforming baselines on challenging tasks in simulation, and further improvement is achieved by incorporating uncertainty estimation.

Title: Irregularity-Informed Time Series Analysis: Adaptive Modelling of Spatial and Temporal Dynamics

• Authors: Liangwei Nathan Zheng, Zhengyang Li, Chang George Dong, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, Weitong Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12257
• Pdf URL: https://arxiv.org/pdf/2410.12257
• Copy Paste: [[2410.12257]] Irregularity-Informed Time Series Analysis: Adaptive Modelling of Spatial and Temporal Dynamics(https://arxiv.org/abs/2410.12257)
• Keywords: transformer
• Abstract: Irregular Time Series Data (IRTS) has shown increasing prevalence in real-world applications. We observed that IRTS can be divided into two specialized types: Natural Irregular Time Series (NIRTS) and Accidental Irregular Time Series (AIRTS). Various existing methods either ignore the impacts of irregular patterns or statically learn the irregular dynamics of NIRTS and AIRTS data and suffer from limited data availability due to the sparsity of IRTS. We proposed a novel transformer-based framework for general irregular time series data that treats IRTS from four views: Locality, Time, Spatio and Irregularity to motivate the data usage to the highest potential. Moreover, we design a sophisticated irregularity-gate mechanism to adaptively select task-relevant information from irregularity, which improves the generalization ability to various IRTS data. We implement extensive experiments to demonstrate the resistance of our work to three highly missing ratio datasets (88.4\%, 94.9\%, 60\% missing value) and investigate the significance of the irregularity information for both NIRTS and AIRTS by additional ablation study. We release our implementation in this https URL

Title: Game Theory Meets Statistical Mechanics in Deep Learning Design

• Authors: Djamel Bouchaffra, Fayçal Ykhlef, Bilal Faye, Hanane Azzag, Mustapha Lebbah
• Subjects: cs.LG, cs.GT
• Abstract URL: https://arxiv.org/abs/2410.12264
• Pdf URL: https://arxiv.org/pdf/2410.12264
• Copy Paste: [[2410.12264]] Game Theory Meets Statistical Mechanics in Deep Learning Design(https://arxiv.org/abs/2410.12264)
• Keywords: extraction
• Abstract: We present a novel deep graphical representation that seamlessly merges principles of game theory with laws of statistical mechanics. It performs feature extraction, dimensionality reduction, and pattern classification within a single learning framework. Our approach draws an analogy between neurons in a network and players in a game theory model. Furthermore, each neuron viewed as a classical particle (subject to statistical physics' laws) is mapped to a set of actions representing specific activation value, and neural network layers are conceptualized as games in a sequential cooperative game theory setting. The feed-forward process in deep learning is interpreted as a sequential game, where each game comprises a set of players. During training, neurons are iteratively evaluated and filtered based on their contributions to a payoff function, which is quantified using the Shapley value driven by an energy function. Each set of neurons that significantly contributes to the payoff function forms a strong coalition. These neurons are the only ones permitted to propagate the information forward to the next layers. We applied this methodology to the task of facial age estimation and gender classification. Experimental results demonstrate that our approach outperforms both multi-layer perceptron and convolutional neural network models in terms of efficiency and accuracy.

Title: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation

• Authors: Junjie Chen, Weihang Su, Zhumin Chu, Haitao Li, Qinyao Ai, Yiqun Liu, Min Zhang, Shaoping Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12265
• Pdf URL: https://arxiv.org/pdf/2410.12265
• Copy Paste: [[2410.12265]] An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation(https://arxiv.org/abs/2410.12265)
• Keywords: large language model
• Abstract: With the rapid development of large language models (LLMs), how to efficiently evaluate them has become an important research question. Existing evaluation methods often suffer from high costs, limited test formats, the need of human references, and systematic evaluation biases. To address these limitations, our study introduces the Auto-PRE, an automatic LLM evaluation framework based on peer review. In contrast to previous studies that rely on human annotations, Auto-PRE selects evaluator LLMs automatically based on their inherent traits including consistency, self-confidence, and pertinence. We conduct extensive experiments on three tasks: summary generation, non-factoid question-answering, and dialogue generation. Experimental results indicate our Auto-PRE achieves state-of-the-art performance at a lower cost. Moreover, our study highlights the impact of prompt strategies and evaluation formats on evaluation performance, offering guidance for method optimization in the future.

Title: DaDiff: Domain-aware Diffusion Model for Nighttime UAV Tracking

• Authors: Haobo Zuo, Changhong Fu, Guangze Zheng, Liangliang Yao, Kunhan Lu, Jia Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12270
• Pdf URL: https://arxiv.org/pdf/2410.12270
• Copy Paste: [[2410.12270]] DaDiff: Domain-aware Diffusion Model for Nighttime UAV Tracking(https://arxiv.org/abs/2410.12270)
• Keywords: robust, diffusion
• Abstract: Domain adaptation is an inspiring solution to the misalignment issue of day/night image features for nighttime UAV tracking. However, the one-step adaptation paradigm is inadequate in addressing the prevalent difficulties posed by low-resolution (LR) objects when viewed from the UAVs at night, owing to the blurry edge contour and limited detail information. Moreover, these approaches struggle to perceive LR objects disturbed by nighttime noise. To address these challenges, this work proposes a novel progressive alignment paradigm, named domain-aware diffusion model (DaDiff), aligning nighttime LR object features to the daytime by virtue of progressive and stable generations. The proposed DaDiff includes an alignment encoder to enhance the detail information of nighttime LR objects, a tracking-oriented layer designed to achieve close collaboration with tracking tasks, and a successive distribution discriminator presented to distinguish different feature distributions at each diffusion timestep successively. Furthermore, an elaborate nighttime UAV tracking benchmark is constructed for LR objects, namely NUT-LR, consisting of 100 annotated sequences. Exhaustive experiments have demonstrated the robustness and feature alignment ability of the proposed DaDiff. The source code and video demo are available at this https URL.

Title: Kallini et al. (2024) do not compare impossible languages with constituency-based ones

• Authors: Tim Hunter
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12271
• Pdf URL: https://arxiv.org/pdf/2410.12271
• Copy Paste: [[2410.12271]] Kallini et al. (2024) do not compare impossible languages with constituency-based ones(https://arxiv.org/abs/2410.12271)
• Keywords: large language model
• Abstract: A central goal of linguistic theory is to find a precise characterization of the notion "possible human language", in the form of a computational device that is capable of describing all and only the languages that can be acquired by a typically developing human child. The success of recent large language models (LLMs) in NLP applications arguably raises the possibility that LLMs might be computational devices that meet this goal. This would only be the case if, in addition to succeeding in learning human languages, LLMs struggle to learn "impossible" human languages. Kallini et al. (2024; "Mission: Impossible Language Models", Proc. ACL) conducted experiments aiming to test this by training GPT-2 on a variety of synthetic languages, and found that it learns some more successfully than others. They present these asymmetries as support for the idea that LLMs' inductive biases align with what is regarded as "possible" for human languages, but the most significant comparison has a confound that makes this conclusion unwarranted. In this paper I explain the confound and suggest some ways forward towards constructing a comparison that appropriately tests the underlying issue.

Title: Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond

• Authors: Pengwei Liang, Junjun Jiang, Qing Ma, Xianming Liu, Jiayi Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12274
• Pdf URL: https://arxiv.org/pdf/2410.12274
• Copy Paste: [[2410.12274]] Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond(https://arxiv.org/abs/2410.12274)
• Keywords: robust, segmentation
• Abstract: Image fusion is famous as an alternative solution to generate one high-quality image from multiple images in addition to image restoration from a single degraded image. The essence of image fusion is to integrate complementary information from source images. Existing fusion methods struggle with generalization across various tasks and often require labor-intensive designs, in which it is difficult to identify and extract useful information from source images due to the diverse requirements of each fusion task. Additionally, these methods develop highly specialized features for different downstream applications, hindering the adaptation to new and diverse downstream tasks. To address these limitations, we introduce DeFusion++, a novel framework that leverages self-supervised learning (SSL) to enhance the versatility of feature representation for different image fusion tasks. DeFusion++ captures the image fusion task-friendly representations from large-scale data in a self-supervised way, overcoming the constraints of limited fusion datasets. Specifically, we introduce two innovative pretext tasks: common and unique decomposition (CUD) and masked feature modeling (MFM). CUD decomposes source images into abstract common and unique components, while MFM refines these components into robust fused features. Jointly training of these tasks enables DeFusion++ to produce adaptable representations that can effectively extract useful information from various source images, regardless of the fusion task. The resulting fused representations are also highly adaptable for a wide range of downstream tasks, including image segmentation and object detection. DeFusion++ stands out by producing versatile fused representations that can enhance both the quality of image fusion and the effectiveness of downstream high-level vision tasks, simplifying the process with the elegant fusion framework.

Title: Controlled Automatic Task-Specific Synthetic Data Generation for Hallucination Detection

• Authors: Yong Xie, Karan Aggarwal, Aitzaz Ahmad, Stephen Lau
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12278
• Pdf URL: https://arxiv.org/pdf/2410.12278
• Copy Paste: [[2410.12278]] Controlled Automatic Task-Specific Synthetic Data Generation for Hallucination Detection(https://arxiv.org/abs/2410.12278)
• Keywords: robust
• Abstract: We present a novel approach to automatically generate non-trivial task-specific synthetic datasets for hallucination detection. Our approach features a two-step generation-selection pipeline, using hallucination pattern guidance and a language style alignment during generation. Hallucination pattern guidance leverages the most important task-specific hallucination patterns while language style alignment aligns the style of the synthetic dataset with benchmark text. To obtain robust supervised detectors from synthetic datasets, we also adopt a data mixture strategy to improve performance robustness and generalization. Our results on three datasets show that our generated hallucination text is more closely aligned with non-hallucinated text versus baselines, to train hallucination detectors with better generalization. Our hallucination detectors trained on synthetic datasets outperform in-context-learning (ICL)-based detectors by a large margin of 32%. Our extensive experiments confirm the benefits of our approach with cross-task and cross-generator generalization. Our data-mixture-based training further improves the generalization and robustness of hallucination detection.

Title: How much do contextualized representations encode long-range context?

• Authors: Simeng Sun, Cheng-Ping Hsieh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12292
• Pdf URL: https://arxiv.org/pdf/2410.12292
• Copy Paste: [[2410.12292]] How much do contextualized representations encode long-range context?(https://arxiv.org/abs/2410.12292)
• Keywords: transformer
• Abstract: We analyze contextual representations in neural autoregressive language models, emphasizing long-range contexts that span several thousand tokens. Our methodology employs a perturbation setup and the metric \emph{Anisotropy-Calibrated Cosine Similarity}, to capture the degree of contextualization of long-range patterns from the perspective of representation geometry. We begin the analysis with a case study on standard decoder-only Transformers, demonstrating that similar perplexity can exhibit markedly different downstream task performance, which can be explained by the difference in contextualization of long-range content. Next, we extend the analysis to other models, covering recent novel architectural designs and various training configurations. The representation-level results illustrate a reduced capacity for high-complexity (i.e., less compressible) sequences across architectures, and that fully recurrent models rely heavily on local context, whereas hybrid models more effectively encode the entire sequence structure. Finally, preliminary analysis of model size and training configurations on the encoding of long-range context suggest potential directions for improving existing language models.

Title: Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors

• Authors: Linwei Tao, Haolan Guo, Minjing Dong, Chang Xu
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12295
• Pdf URL: https://arxiv.org/pdf/2410.12295
• Copy Paste: [[2410.12295]] Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors(https://arxiv.org/abs/2410.12295)
• Keywords: large language model
• Abstract: Calibration is crucial in deep learning applications, especially in fields like healthcare and autonomous driving, where accurate confidence estimates are vital for decision-making. However, deep neural networks often suffer from miscalibration, with reliability diagrams and Expected Calibration Error (ECE) being the only standard perspective for evaluating calibration performance. In this paper, we introduce the concept of consistency as an alternative perspective on model calibration, inspired by uncertainty estimation literature in large language models (LLMs). We highlight its advantages over the traditional reliability-based view. Building on this concept, we propose a post-hoc calibration method called Consistency Calibration (CC), which adjusts confidence based on the model's consistency across perturbed inputs. CC is particularly effective in locally uncertainty estimation, as it requires no additional data samples or label information, instead generating input perturbations directly from the source data. Moreover, we show that performing perturbations at the logit level significantly improves computational efficiency. We validate the effectiveness of CC through extensive comparisons with various post-hoc and training-time calibration methods, demonstrating state-of-the-art performance on standard datasets such as CIFAR-10, CIFAR-100, and ImageNet, as well as on long-tailed datasets like ImageNet-LT.

Title: Pyramid-Driven Alignment: Pyramid Principle Guided Integration of Large Language Models and Knowledge Graphs

• Authors: Lei Sun, Xinchen Wang, Youdi Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12298
• Pdf URL: https://arxiv.org/pdf/2410.12298
• Copy Paste: [[2410.12298]] Pyramid-Driven Alignment: Pyramid Principle Guided Integration of Large Language Models and Knowledge Graphs(https://arxiv.org/abs/2410.12298)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) possess impressive reasoning abilities but are prone to generating incorrect information, often referred to as hallucinations. While incorporating external Knowledge Graphs (KGs) can partially mitigate this issue, existing methods primarily treat KGs as static knowledge repositories, overlooking the critical disparity between KG and LLM knowledge, and failing to fully exploit the reasoning capabilities inherent in KGs. To address these limitations, we propose Pyramid-Driven Alignment (PDA), a novel framework for seamlessly integrating LLMs with KGs. PDA utilizes Pyramid Principle analysis to construct a hierarchical pyramid structure. This structure is designed to reflect the input question and generate more validated deductive knowledge, thereby enhancing the alignment of LLMs and KGs and ensuring more cohesive integration. Furthermore, PDA employs a recursive mechanism to harness the underlying reasoning abilities of KGs, resulting in more accurate knowledge retrieval for question-answering tasks. Our experimental results reveal a substantial performance advantage of PDA over state-of-the-art baselines, with improvements reaching 26.70% and 26.78%.

Title: Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors

• Authors: Weixuan Wang, Jingyuan Yang, Wei Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12299
• Pdf URL: https://arxiv.org/pdf/2410.12299
• Copy Paste: [[2410.12299]] Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors(https://arxiv.org/abs/2410.12299)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved remarkable performance across many tasks, yet aligning them with desired behaviors remains challenging. Activation intervention has emerged as an effective and economical method to modify the behavior of LLMs. Despite considerable interest in this area, current intervention methods exclusively employ a fixed steering vector to modify model activations, lacking adaptability to diverse input semantics. To address this limitation, we propose Semantics-Adaptive Dynamic Intervention (SADI), a novel method that constructs a dynamic steering vector to intervene model activations at inference time. More specifically, SADI utilizes activation differences in contrastive pairs to precisely identify critical elements of an LLM (i.e., attention heads, hidden states, and neurons) for targeted intervention. During inference, SADI dynamically steers model behavior by scaling element-wise activations based on the directions of input semantics. Experimental results show that SADI outperforms established baselines by substantial margins, improving task performance without training. SADI's cost-effectiveness and generalizability across various LLM backbones and tasks highlight its potential as a versatile alignment technique. In addition, we release the code to foster research along this line:this https URL.

Title: DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domain

• Authors: Fengpeng Li, Kemou Li, Haiwei Wu, Jinyu Tian, Jiantao Zhou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12307
• Pdf URL: https://arxiv.org/pdf/2410.12307
• Copy Paste: [[2410.12307]] DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domain(https://arxiv.org/abs/2410.12307)
• Keywords: protect, attack, robust, generative
• Abstract: To protect deep neural networks (DNNs) from adversarial attacks, adversarial training (AT) is developed by incorporating adversarial examples (AEs) into model training. Recent studies show that adversarial attacks disproportionately impact the patterns within the phase of the sample's frequency spectrum -- typically containing crucial semantic information -- more than those in the amplitude, resulting in the model's erroneous categorization of AEs. We find that, by mixing the amplitude of training samples' frequency spectrum with those of distractor images for AT, the model can be guided to focus on phase patterns unaffected by adversarial perturbations. As a result, the model's robustness can be improved. Unfortunately, it is still challenging to select appropriate distractor images, which should mix the amplitude without affecting the phase patterns. To this end, in this paper, we propose an optimized Adversarial Amplitude Generator (AAG) to achieve a better tradeoff between improving the model's robustness and retaining phase patterns. Based on this generator, together with an efficient AE production procedure, we design a new Dual Adversarial Training (DAT) strategy. Experiments on various datasets show that our proposed DAT leads to significantly improved robustness against diverse adversarial attacks.

Title: Correction to Local Information Privacy and Its Applications to Data Aggregation

• Authors: Bo Jiang, Ming Li, Ravi Tandon
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.12309
• Pdf URL: https://arxiv.org/pdf/2410.12309
• Copy Paste: [[2410.12309]] Correction to Local Information Privacy and Its Applications to Data Aggregation(https://arxiv.org/abs/2410.12309)
• Keywords: privacy
• Abstract: In our previous works, we defined Local Information Privacy (LIP) as a context-aware privacy notion and presented the corresponding privacy-preserving mechanism. Then we claim that the mechanism satisfies epsilon-LIP for any epsilon>0 for arbitrary Px. However, this claim is not completely correct. In this document, we provide a correction to the valid range of privacy parameters of our previously proposed LIP mechanism. Further, we propose efficient algorithms to expand the range of valid privacy parameters. Finally, we discuss the impact of updated results on our original paper's experiments, the rationale of the proposed correction and corrected results.

Title: Open Domain Question Answering with Conflicting Contexts

• Authors: Siyi Liu, Qiang Ning, Kishaloy Halder, Wei Xiao, Zheng Qi, Phu Mon Htut, Yi Zhang, Neha Anna John, Bonan Min, Yassine Benajiba, Dan Roth
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12311
• Pdf URL: https://arxiv.org/pdf/2410.12311
• Copy Paste: [[2410.12311]] Open Domain Question Answering with Conflicting Contexts(https://arxiv.org/abs/2410.12311)
• Keywords: large language model
• Abstract: Open domain question answering systems frequently rely on information retrieved from large collections of text (such as the Web) to answer questions. However, such collections of text often contain conflicting information, and indiscriminately depending on this information may result in untruthful and inaccurate answers. To understand the gravity of this problem, we collect a human-annotated dataset, Question Answering with Conflicting Contexts (QACC), and find that as much as 25% of unambiguous, open domain questions can lead to conflicting contexts when retrieved using Google Search. We evaluate and benchmark three powerful Large Language Models (LLMs) with our dataset QACC and demonstrate their limitations in effectively addressing questions with conflicting information. To explore how humans reason through conflicting contexts, we request our annotators to provide explanations for their selections of correct answers. We demonstrate that by finetuning LLMs to explain their answers, we can introduce richer information into their training that guide them through the process of reasoning with conflicting contexts.

Title: FaceChain-FACT: Face Adapter with Decoupled Training for Identity-preserved Personalization

• Authors: Cheng Yu, Haoyu Xie, Lei Shang, Yang Liu, Jun Dan, Baigui Sun, Liefeng Bo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12312
• Pdf URL: https://arxiv.org/pdf/2410.12312
• Copy Paste: [[2410.12312]] FaceChain-FACT: Face Adapter with Decoupled Training for Identity-preserved Personalization(https://arxiv.org/abs/2410.12312)
• Keywords: extraction, fair, transformer, generative
• Abstract: In the field of human-centric personalized image generation, the adapter-based method obtains the ability to customize and generate portraits by text-to-image training on facial data. This allows for identity-preserved personalization without additional fine-tuning in inference. Although there are improvements in efficiency and fidelity, there is often a significant performance decrease in test following ability, controllability, and diversity of generated faces compared to the base model. In this paper, we analyze that the performance degradation is attributed to the failure to decouple identity features from other attributes during extraction, as well as the failure to decouple the portrait generation training from the overall generation task. To address these issues, we propose the Face Adapter with deCoupled Training (FACT) framework, focusing on both model architecture and training strategy. To decouple identity features from others, we leverage a transformer-based face-export encoder and harness fine-grained identity features. To decouple the portrait generation training, we propose Face Adapting Increment Regularization~(FAIR), which effectively constrains the effect of face adapters on the facial region, preserving the generative ability of the base model. Additionally, we incorporate a face condition drop and shuffle mechanism, combined with curriculum learning, to enhance facial controllability and diversity. As a result, FACT solely learns identity preservation from training data, thereby minimizing the impact on the original text-to-image capabilities of the base model. Extensive experiments show that FACT has both controllability and fidelity in both text-to-image generation and inpainting solutions for portrait generation.

Title: TPFL: A Trustworthy Personalized Federated Learning Framework via Subjective Logic

• Authors: Jinqian Chen, Jihua Zhu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.12316
• Pdf URL: https://arxiv.org/pdf/2410.12316
• Copy Paste: [[2410.12316]] TPFL: A Trustworthy Personalized Federated Learning Framework via Subjective Logic(https://arxiv.org/abs/2410.12316)
• Keywords: secure, privacy, protect, attack, robust, federate
• Abstract: Federated learning (FL) enables collaborative model training across distributed clients while preserving data privacy. Despite its widespread adoption, most FL approaches focusing solely on privacy protection fall short in scenarios where trustworthiness is crucial, necessitating advancements in secure training, dependable decision-making mechanisms, robustness on corruptions, and enhanced performance with Non-IID data. To bridge this gap, we introduce Trustworthy Personalized Federated Learning (TPFL) framework designed for classification tasks via subjective logic in this paper. Specifically, TPFL adopts a unique approach by employing subjective logic to construct federated models, providing probabilistic decisions coupled with an assessment of uncertainty rather than mere probability assignments. By incorporating a trainable heterogeneity prior to the local training phase, TPFL effectively mitigates the adverse effects of data heterogeneity. Model uncertainty and instance uncertainty are further utilized to ensure the safety and reliability of the training and inference stages. Through extensive experiments on widely recognized federated learning benchmarks, we demonstrate that TPFL not only achieves competitive performance compared with advanced methods but also exhibits resilience against prevalent malicious attacks, robustness on domain shifts, and reliability in high-stake scenarios.

Title: UTF:Undertrained Tokens as Fingerprints A Novel Approach to LLM Identification

• Authors: Jiacheng Cai, Jiahao Yu, Yangguang Shao, Yuhang Wu, Xinyu Xing
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12318
• Pdf URL: https://arxiv.org/pdf/2410.12318
• Copy Paste: [[2410.12318]] UTF:Undertrained Tokens as Fingerprints A Novel Approach to LLM Identification(https://arxiv.org/abs/2410.12318)
• Keywords: robust, large language model
• Abstract: Fingerprinting large language models (LLMs) is essential for verifying model ownership, ensuring authenticity, and preventing misuse. Traditional fingerprinting methods often require significant computational overhead or white-box verification access. In this paper, we introduce UTF, a novel and efficient approach to fingerprinting LLMs by leveraging under-trained tokens. Under-trained tokens are tokens that the model has not fully learned during its training phase. By utilizing these tokens, we perform supervised fine-tuning to embed specific input-output pairs into the model. This process allows the LLM to produce predetermined outputs when presented with certain inputs, effectively embedding a unique fingerprint. Our method has minimal overhead and impact on model's performance, and does not require white-box access to target model's ownership identification. Compared to existing fingerprinting methods, UTF is also more effective and robust to fine-tuning and random guess.

Title: Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up

• Authors: Jiahao Yuan, Dehui Du, Hao Zhang, Zixiang Di, Usman Naseem
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12323
• Pdf URL: https://arxiv.org/pdf/2410.12323
• Copy Paste: [[2410.12323]] Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up(https://arxiv.org/abs/2410.12323)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable performance in reasoning tasks but face limitations in mathematical and complex logical reasoning. Existing methods to improve LLMs' logical capabilities either involve traceable or verifiable logical sequences that generate more reliable responses by constructing logical structures yet increase computational costs, or introduces rigid logic template rules, reducing flexibility. In this paper, we propose Reversal of Thought (RoT), a novel framework aimed at enhancing the logical reasoning abilities of LLMs. RoT utilizes a Preference-Guided Reverse Reasoning warm-up strategy, which integrates logical symbols for pseudocode planning through meta-cognitive mechanisms and pairwise preference self-evaluation to generate task-specific prompts solely through demonstrations, aligning with LLMs' cognitive preferences shaped by Reinforcement Learning with Human Feedback (RLHF). Through reverse reasoning, we ultilize a Cognitive Preference Manager to assess knowledge boundaries and further expand LLMs' reasoning capabilities by aggregating solution logic for known tasks and stylistic templates for unknown tasks. Experiments across various tasks demonstrate that RoT surpasses existing baselines in both reasoning accuracy and efficiency.

Title: Optimizing Low-Resource Language Model Training: Comprehensive Analysis of Multi-Epoch, Multi-Lingual, and Two-Stage Approaches

• Authors: Kosuke Akimoto, Masafumi Oyamada
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12325
• Pdf URL: https://arxiv.org/pdf/2410.12325
• Copy Paste: [[2410.12325]] Optimizing Low-Resource Language Model Training: Comprehensive Analysis of Multi-Epoch, Multi-Lingual, and Two-Stage Approaches(https://arxiv.org/abs/2410.12325)
• Keywords: large language model
• Abstract: In this paper, we address the challenge of optimizing training setups for Large Language Models (LLMs) of low-resource language with a limited amount of corpus. Existing works adopt multi-epoch, multi-lingual, and two-stage training to utilize the limited target language corpus efficiently. However, there is still a lack of understanding about the optimal hyperparameter setups for combining these three approaches to train LLMs. We exhaustively explore training setups for low-resource language LLM, combining these three approaches, and found the following insights for efficiently reducing the cost of hyperparameter search: (1) As the amount of target language corpus decreases, the optimal training approach shifts from monolingual single-stage training to multi-lingual two-stage training at a compute budget dependent threshold. (2) The optimal model scale remains stable regardless of the amount of target language corpus, allowing the use of the compute-optimal scale of monolingual training. (3) The optimal number of epochs can be extrapolated from smaller-scale experiments to larger scale using our proposed model. Also, we provide evidence that, in single-stage training, the target language validation loss follows a power law with respect to the target language ratio, with an exponent independent of the amount of data, model scale, and language pair.

Title: Revisited Large Language Model for Time Series Analysis through Modality Alignment

• Authors: Liangwei Nathan Zheng, Chang George Dong, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, Weitong Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12326
• Pdf URL: https://arxiv.org/pdf/2410.12326
• Copy Paste: [[2410.12326]] Revisited Large Language Model for Time Series Analysis through Modality Alignment(https://arxiv.org/abs/2410.12326)
• Keywords: large language model
• Abstract: Large Language Models have demonstrated impressive performance in many pivotal web applications such as sensor data analysis. However, since LLMs are not designed for time series tasks, simpler models like linear regressions can often achieve comparable performance with far less complexity. In this study, we perform extensive experiments to assess the effectiveness of applying LLMs to key time series tasks, including forecasting, classification, imputation, and anomaly detection. We compare the performance of LLMs against simpler baseline models, such as single-layer linear models and randomly initialized LLMs. Our results reveal that LLMs offer minimal advantages for these core time series tasks and may even distort the temporal structure of the data. In contrast, simpler models consistently outperform LLMs while requiring far fewer parameters. Furthermore, we analyze existing reprogramming techniques and show, through data manifold analysis, that these methods fail to effectively align time series data with language and display pseudo-alignment behaviour in embedding space. Our findings suggest that the performance of LLM-based methods in time series tasks arises from the intrinsic characteristics and structure of time series data, rather than any meaningful alignment with the language model architecture.

Title: Neuron-based Personality Trait Induction in Large Language Models

• Authors: Jia Deng, Tianyi Tang, Yanbin Yin, Wenhao Yang, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12327
• Pdf URL: https://arxiv.org/pdf/2410.12327
• Copy Paste: [[2410.12327]] Neuron-based Personality Trait Induction in Large Language Models(https://arxiv.org/abs/2410.12327)
• Keywords: generative, large language model
• Abstract: Large language models (LLMs) have become increasingly proficient at simulating various personality traits, an important capability for supporting related applications (e.g., role-playing). To further improve this capacity, in this paper, we present a neuron-based approach for personality trait induction in LLMs, with three major technical contributions. First, we construct PersonalityBench, a large-scale dataset for identifying and evaluating personality traits in LLMs. This dataset is grounded in the Big Five personality traits from psychology and is designed to assess the generative capabilities of LLMs towards specific personality traits. Second, by leveraging PersonalityBench, we propose an efficient method for identifying personality-related neurons within LLMs by examining the opposite aspects of a given trait. Third, we develop a simple yet effective induction method that manipulates the values of these identified personality-related neurons. This method enables fine-grained control over the traits exhibited by LLMs without training and modifying model parameters. Extensive experiments validate the efficacy of our neuron identification and trait induction methods. Notably, our approach achieves comparable performance as fine-tuned models, offering a more efficient and flexible solution for personality trait induction in LLMs. We provide access to all the mentioned resources at this https URL.

Title: Understanding the Role of LLMs in Multimodal Evaluation Benchmarks

• Authors: Botian Jiang, Lei Li, Xiaonan Li, Zhaowei Li, Xiachong Feng, Lingpeng Kong, Qi Liu, Xipeng Qiu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12329
• Pdf URL: https://arxiv.org/pdf/2410.12329
• Copy Paste: [[2410.12329]] Understanding the Role of LLMs in Multimodal Evaluation Benchmarks(https://arxiv.org/abs/2410.12329)
• Keywords: large language model
• Abstract: The rapid advancement of Multimodal Large Language Models (MLLMs) has been accompanied by the development of various benchmarks to evaluate their capabilities. However, the true nature of these evaluations and the extent to which they assess multimodal reasoning versus merely leveraging the underlying Large Language Model (LLM) backbone remain unclear. This paper presents a comprehensive investigation into the role of LLM backbones in MLLM evaluation, focusing on two critical aspects: the degree to which current benchmarks truly assess multimodal reasoning and the influence of LLM prior knowledge on performance. Specifically, we introduce a modified evaluation protocol to disentangle the contributions of the LLM backbone from multimodal integration, and an automatic knowledge identification technique for diagnosing whether LLMs equip the necessary knowledge for corresponding multimodal questions. Our study encompasses four diverse MLLM benchmarks and eight state-of-the-art MLLMs. Key findings reveal that some benchmarks allow high performance even without visual inputs and up to 50\% of error rates can be attributed to insufficient world knowledge in the LLM backbone, indicating a heavy reliance on language capabilities. To address knowledge deficiencies, we propose a knowledge augmentation pipeline that achieves significant performance gains, with improvements of up to 60\% on certain datasets, resulting in a approximately 4x increase in performance. Our work provides crucial insights into the role of the LLM backbone in MLLMs, and highlights the need for more nuanced benchmarking approaches.

Title: MAX: Masked Autoencoder for X-ray Fluorescence in Geological Investigation

• Authors: An-Sheng Lee, Yu-Wen Pao, Hsuan-Tien Lin, Sofia Ya Hsuan Liou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12330
• Pdf URL: https://arxiv.org/pdf/2410.12330
• Copy Paste: [[2410.12330]] MAX: Masked Autoencoder for X-ray Fluorescence in Geological Investigation(https://arxiv.org/abs/2410.12330)
• Keywords: robust, explainability
• Abstract: Pre-training foundation models has become the de-facto procedure for deep learning approaches, yet its application remains limited in the geological studies, where in needs of the model transferability to break the shackle of data scarcity. Here we target on the X-ray fluorescence (XRF) scanning data, a standard high-resolution measurement in extensive scientific drilling projects. We propose a scalable self-supervised learner, masked autoencoders on XRF spectra (MAX), to pre-train a foundation model covering geological records from multiple regions of the Pacific and Southern Ocean. In pre-training, we find that masking a high proportion of the input spectrum (50\%) yields a nontrivial and meaningful self-supervisory task. For downstream tasks, we select the quantification of XRF spectra into two costly geochemical measurements, CaCO$_3$ and total organic carbon, due to their importance in understanding the paleo-oceanic carbon system. Our results show that MAX, requiring only one-third of the data, outperforms models without pre-training in terms of quantification accuracy. Additionally, the model's generalizability improves by more than 60\% in zero-shot tests on new materials, with explainability further ensuring its robustness. Thus, our approach offers a promising pathway to overcome data scarcity in geological discovery by leveraging the self-supervised foundation model and fast-acquired XRF scanning data.

Title: MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs

• Authors: Yunqiu Xu, Linchao Zhu, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12332
• Pdf URL: https://arxiv.org/pdf/2410.12332
• Copy Paste: [[2410.12332]] MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs(https://arxiv.org/abs/2410.12332)
• Keywords: large language model
• Abstract: While multimodal large language models (MLLMs) have demonstrated extraordinary vision-language understanding capabilities and shown potential to serve as general-purpose assistants, their abilities to solve instance-level visual-language problems beyond a single image warrant further exploration. In order to assess these unproven abilities of MLLMs, this paper proposes a new visual grounding task called multi-context visual grounding, which aims to localize instances of interest across multiple images based on open-ended text prompts. To facilitate this research, we meticulously construct a new dataset MC-Bench for benchmarking the visual grounding capabilities of MLLMs. MC-Bench features 2K high-quality and manually annotated samples, consisting of instance-level labeled image pairs and corresponding text prompts that indicate the target instances in the images. In total, there are three distinct styles of text prompts, covering 20 practical skills. We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities. Our evaluation reveals a non-trivial performance gap between existing MLLMs and humans across all metrics. We also observe that existing MLLMs typically outperform foundation models without LLMs only on image-level metrics, and the specialist MLLMs trained on single images often struggle to generalize to multi-image scenarios. Moreover, a simple stepwise baseline integrating advanced MLLM and a detector can significantly surpass prior end-to-end MLLMs. We hope our MC-Bench and empirical findings can encourage the research community to further explore and enhance the untapped potentials of MLLMs in instance-level tasks, particularly in multi-image contexts. Project page: this https URL.

Title: A linguistic analysis of undesirable outcomes in the era of generative AI

• Authors: Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, Luca Pappalardo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12341
• Pdf URL: https://arxiv.org/pdf/2410.12341
• Copy Paste: [[2410.12341]] A linguistic analysis of undesirable outcomes in the era of generative AI(https://arxiv.org/abs/2410.12341)
• Keywords: generative
• Abstract: Recent research has focused on the medium and long-term impacts of generative AI, posing scientific and societal challenges mainly due to the detection and reliability of machine-generated information, which is projected to form the major content on the Web soon. Prior studies show that LLMs exhibit a lower performance in generation tasks (model collapse) as they undergo a fine-tuning process across multiple generations on their own generated content (self-consuming loop). In this paper, we present a comprehensive simulation framework built upon the chat version of LLama2, focusing particularly on the linguistic aspects of the generated content, which has not been fully examined in existing studies. Our results show that the model produces less lexical rich content across generations, reducing diversity. The lexical richness has been measured using the linguistic measures of entropy and TTR as well as calculating the POSTags frequency. The generated content has also been examined with an $n$-gram analysis, which takes into account the word order, and semantic networks, which consider the relation between different words. These findings suggest that the model collapse occurs not only by decreasing the content diversity but also by distorting the underlying linguistic patterns of the generated text, which both highlight the critical importance of carefully choosing and curating the initial input text, which can alleviate the model collapse problem. Furthermore, we conduct a qualitative analysis of the fine-tuned models of the pipeline to compare their performances on generic NLP tasks to the original model. We find that autophagy transforms the initial model into a more creative, doubtful and confused one, which might provide inaccurate answers and include conspiracy theories in the model responses, spreading false and biased information on the Web.

Title: Federated Temporal Graph Clustering

• Authors: Yang Liu, Zihao Zhou, Xianghong Xu, Qian Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.12343
• Pdf URL: https://arxiv.org/pdf/2410.12343
• Copy Paste: [[2410.12343]] Federated Temporal Graph Clustering(https://arxiv.org/abs/2410.12343)
• Keywords: privacy, federate
• Abstract: Temporal graph clustering is a complex task that involves discovering meaningful structures in dynamic graphs where relationships and entities change over time. Existing methods typically require centralized data collection, which poses significant privacy and communication challenges. In this work, we introduce a novel Federated Temporal Graph Clustering (FTGC) framework that enables decentralized training of graph neural networks (GNNs) across multiple clients, ensuring data privacy throughout the process. Our approach incorporates a temporal aggregation mechanism to effectively capture the evolution of graph structures over time and a federated optimization strategy to collaboratively learn high-quality clustering representations. By preserving data privacy and reducing communication overhead, our framework achieves competitive performance on temporal graph datasets, making it a promising solution for privacy-sensitive, real-world applications involving dynamic data.

Title: Towards Flexible and Efficient Diffusion Low Light Enhancer

• Authors: Guanzhou Lan, Qianli Ma, Yuqi Yang, Zhigang Wang, Dong Wang, Yuan Yuan, Bin Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12346
• Pdf URL: https://arxiv.org/pdf/2410.12346
• Copy Paste: [[2410.12346]] Towards Flexible and Efficient Diffusion Low Light Enhancer(https://arxiv.org/abs/2410.12346)
• Keywords: diffusion
• Abstract: Diffusion-based Low-Light Image Enhancement (LLIE) has demonstrated significant success in improving the visibility of low-light images. However, the substantial computational burden introduced by the iterative sampling process remains a major concern. Current acceleration methods, whether training-based or training-free, often lead to significant performance degradation. As a result, to achieve an efficient student model with performance comparable to that of existing multi-step teacher model, it is usually necessary to retrain a more capable teacher model. This approach introduces inflexibility, as it requires additional training to enhance the teacher's performance. To address these challenges, we propose \textbf{Re}flectance-aware \textbf{D}iffusion with \textbf{Di}stilled \textbf{T}rajectory (\textbf{ReDDiT}), a step distillation framework specifically designed for LLIE. ReDDiT trains a student model to replicate the teacher's trajectory in fewer steps while also possessing the ability to surpass the teacher's performance. Specifically, we first introduce a trajectory decoder from the teacher model to provide guidance. Subsequently, a reflectance-aware trajectory refinement module is incorporated into the distillation process to enable more deterministic guidance from the teacher model. Our framework achieves comparable performance to previous diffusion-based methods with redundant steps in just 2 steps while establishing new state-of-the-art (SOTA) results with 8 or 4 steps. Comprehensive experimental evaluations on 10 benchmark datasets validate the effectiveness of our method, consistently outperforming existing SOTA methods.

Title: Yama: Precise Opcode-based Data Flow Analysis for Detecting PHP Applications Vulnerabilities

• Authors: Zhao Jiazhen, Zhu Kailong, Yu Lu, Huang Hui, Lu Yuliang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.12351
• Pdf URL: https://arxiv.org/pdf/2410.12351
• Copy Paste: [[2410.12351]] Yama: Precise Opcode-based Data Flow Analysis for Detecting PHP Applications Vulnerabilities(https://arxiv.org/abs/2410.12351)
• Keywords: attack
• Abstract: Web applications encompass various aspects of daily life, including online shopping, e-learning, and internet banking. Once there is a vulnerability, it can cause severe societal and economic damage. Due to its ease of use, PHP has become the preferred server-side programming language for web applications, making PHP applications a primary target for attackers. Data flow analysis is widely used for vulnerability detection before deploying web applications because of its efficiency. However, the high complexity of the PHP language makes it difficult to achieve precise data flow analysis. In this paper, we present Yama, a context-sensitive and path-sensitive interprocedural data flow analysis method for PHP, designed to detect taint-style vulnerabilities in PHP applications. We have found that the precise semantics and clear control flow of PHP opcodes enable data flow analysis to be more precise and efficient. Leveraging this observation, we established parsing rules for PHP opcodes and implemented a precise understanding of PHP program semantics in Yama. We evaluated Yama from three dimensions: basic data flow analysis capabilities, complex semantic analysis capabilities, and the ability to discover vulnerabilities in real-world applications, demonstrating Yama's advancement in vulnerability detection. Specifically, Yama possesses context-sensitive and path-sensitive interprocedural analysis capabilities, achieving a 99.1% true positive rate in complex semantic analysis experiments related to type inference, dynamic features, and built-in functions. It discovered and reported 38 zero-day vulnerabilities across 24 projects on GitHub with over 1,000 stars each, assigning 34 new CVE IDs. We have released the source code of the prototype implementation and the parsing rules for PHP opcodes to facilitate future research.

Title: Towards Neural Scaling Laws for Time Series Foundation Models

• Authors: Qingren Yao, Chao-Han Huck Yang, Renhe Jiang, Yuxuan Liang, Ming Jin, Shirui Pan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12360
• Pdf URL: https://arxiv.org/pdf/2410.12360
• Copy Paste: [[2410.12360]] Towards Neural Scaling Laws for Time Series Foundation Models(https://arxiv.org/abs/2410.12360)
• Keywords: robust, transformer
• Abstract: Scaling laws offer valuable insights into the design of time series foundation models (TSFMs). However, previous research has largely focused on the scaling laws of TSFMs for in-distribution (ID) data, leaving their out-of-distribution (OOD) scaling behavior and the influence of model architectures less explored. In this work, we examine two common TSFM architectures, encoder-only and decoder-only Transformers, and investigate their scaling behavior on both ID and OOD data. These models are trained and evaluated across varying parameter counts, compute budgets, and dataset sizes. Our experiments reveal that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID settings. We further compare the scaling properties across different architectures, incorporating two state-of-the-art TSFMs as case studies, showing that model architecture plays a significant role in scaling. The encoder-only Transformers demonstrate better scalability than the decoder-only Transformers, while the architectural enhancements in the two advanced TSFMs primarily improve ID performance but reduce OOD scalability. While scaling up TSFMs is expected to drive performance breakthroughs, the lack of a comprehensive understanding of TSFM scaling laws has hindered the development of a robust framework to guide model scaling. We fill this gap in this work by synthesizing our findings and providing practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.

Title: GAN Based Top-Down View Synthesis in Reinforcement Learning Environments

• Authors: Usama Younus, Vinoj Jayasundara, Shivam Mishra, Suleyman Aslan
• Subjects: cs.CV, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.12372
• Pdf URL: https://arxiv.org/pdf/2410.12372
• Copy Paste: [[2410.12372]] GAN Based Top-Down View Synthesis in Reinforcement Learning Environments(https://arxiv.org/abs/2410.12372)
• Keywords: generative
• Abstract: Human actions are based on the mental perception of the environment. Even when all the aspects of an environment are not visible, humans have an internal mental model that can generalize the partially visible scenes to fully constructed and connected views. This internal mental model uses learned abstract representations of spatial and temporal aspects of the environments encountered in the past. Artificial agents in reinforcement learning environments also benefit by learning a representation of the environment from experience. It provides the agent with viewpoints that are not directly visible to it, helping it make better policy decisions. It can also be used to predict the future state of the environment. This project explores learning the top-down view of an RL environment based on the artificial agent's first-person view observations with a generative adversarial network(GAN). The top-down view is useful as it provides a complete overview of the environment by building a map of the entire environment. It provides information about the objects' dimensions and shapes along with their relative positions with one another. Initially, when only a partial observation of the environment is visible to the agent, only a partial top-down view is generated. As the agent explores the environment through a set of actions, the generated top-down view becomes complete. This generated top-down view can assist the agent in deducing better policy decisions. The focus of the project is to learn the top-down view of an RL environment. It doesn't deal with any Reinforcement Learning task.

Title: HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims

• Authors: Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, Kunwoo Park
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.12377
• Pdf URL: https://arxiv.org/pdf/2410.12377
• Copy Paste: [[2410.12377]] HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims(https://arxiv.org/abs/2410.12377)
• Keywords: large language model
• Abstract: To tackle the AVeriTeC shared task hosted by the FEVER-24, we introduce a system that only employs publicly available large language models (LLMs) for each step of automated fact-checking, dubbed the Herd of Open LLMs for verifying real-world claims (HerO). HerO employs multiple LLMs for each step of automated fact-checking. For evidence retrieval, a language model is used to enhance a query by generating hypothetical fact-checking documents. We prompt pretrained and fine-tuned LLMs for question generation and veracity prediction by crafting prompts with retrieved in-context samples. HerO achieved 2nd place on the leaderboard with the AVeriTeC score of 0.57, suggesting the potential of open LLMs for verifying real-world claims. For future research, we make our code publicly available at this https URL.

Title: Evaluation of Attribution Bias in Retrieval-Augmented Large Language Models

• Authors: Amin Abolghasemi, Leif Azzopardi, Seyyed Hadi Hashemi, Maarten de Rijke, Suzan Verberne
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12380
• Pdf URL: https://arxiv.org/pdf/2410.12380
• Copy Paste: [[2410.12380]] Evaluation of Attribution Bias in Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2410.12380)
• Keywords: large language model
• Abstract: Attributing answers to source documents is an approach used to enhance the verifiability of a model's output in retrieval augmented generation (RAG). Prior work has mainly focused on improving and evaluating the attribution quality of large language models (LLMs) in RAG, but this may come at the expense of inducing biases in the attribution of answers. We define and examine two aspects in the evaluation of LLMs in RAG pipelines, namely attribution sensitivity and bias with respect to authorship information. We explicitly inform an LLM about the authors of source documents, instruct it to attribute its answers, and analyze (i) how sensitive the LLM's output is to the author of source documents, and (ii) whether the LLM exhibits a bias towards human-written or AI-generated source documents. We design an experimental setup in which we use counterfactual evaluation to study three LLMs in terms of their attribution sensitivity and bias in RAG pipelines. Our results show that adding authorship information to source documents can significantly change the attribution quality of LLMs by 3% to 18%. Moreover, we show that LLMs can have an attribution bias towards explicit human authorship, which can serve as a competing hypothesis for findings of prior work that shows that LLM-generated content may be preferred over human-written contents. Our findings indicate that metadata of source documents can influence LLMs' trust, and how they attribute their answers. Furthermore, our research highlights attribution bias and sensitivity as a novel aspect of brittleness in LLMs.

Title: HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks

• Authors: Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, Jacky Keung
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12381
• Pdf URL: https://arxiv.org/pdf/2410.12381
• Copy Paste: [[2410.12381]] HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks(https://arxiv.org/abs/2410.12381)
• Keywords: large language model
• Abstract: Coding tasks have been valuable for evaluating Large Language Models (LLMs), as they demand the comprehension of high-level instructions, complex reasoning, and the implementation of functional programs -- core capabilities for advancing Artificial General Intelligence. Despite the progress in Large Multimodal Models (LMMs), which extend LLMs with visual perception and understanding capabilities, there remains a notable lack of coding benchmarks that rigorously assess these models, particularly in tasks that emphasize visual reasoning. To address this gap, we introduce HumanEval-V, a novel and lightweight benchmark specifically designed to evaluate LMMs' visual understanding and reasoning capabilities through code generation. HumanEval-V includes 108 carefully crafted, entry-level Python coding tasks derived from platforms like CodeForces and Stack Overflow. Each task is adapted by modifying the context and algorithmic patterns of the original problems, with visual elements redrawn to ensure distinction from the source, preventing potential data leakage. LMMs are required to complete the code solution based on the provided visual context and a predefined Python function signature outlining the task requirements. Every task is equipped with meticulously handcrafted test cases to ensure a thorough and reliable evaluation of model-generated solutions. We evaluate 19 state-of-the-art LMMs using HumanEval-V, uncovering significant challenges. Proprietary models like GPT-4o achieve only 13% pass@1 and 36.4% pass@10, while open-weight models with 70B parameters score below 4% pass@1. Ablation studies further reveal the limitations of current LMMs in vision reasoning and coding capabilities. These results underscore key areas for future research to enhance LMMs' capabilities. We have open-sourced our code and benchmark at this https URL.

Title: Prompt Compression for Large Language Models: A Survey

• Authors: Zongqian Li, Yinhong Liu, Yixuan Su, Nigel Collier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12388
• Pdf URL: https://arxiv.org/pdf/2410.12388
• Copy Paste: [[2410.12388]] Prompt Compression for Large Language Models: A Survey(https://arxiv.org/abs/2410.12388)
• Keywords: large language model
• Abstract: Leveraging large language models (LLMs) for complex natural language tasks typically requires long-form prompts to convey detailed requirements and information, which results in increased memory usage and inference costs. To mitigate these challenges, multiple efficient methods have been proposed, with prompt compression gaining significant research interest. This survey provides an overview of prompt compression techniques, categorized into hard prompt methods and soft prompt methods. First, the technical approaches of these methods are compared, followed by an exploration of various ways to understand their mechanisms, including the perspectives of attention optimization, Parameter-Efficient Fine-Tuning (PEFT), modality fusion, and new synthetic language. We also examine the downstream adaptations of various prompt compression techniques. Finally, the limitations of current prompt compression methods are analyzed, and several future directions are outlined, such as optimizing the compression encoder, combining hard and soft prompts methods, and leveraging insights from multimodality.

Title: Tracking Universal Features Through Fine-Tuning and Model Merging

• Authors: Niels Horn, Desmond Elliott
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12391
• Pdf URL: https://arxiv.org/pdf/2410.12391
• Copy Paste: [[2410.12391]] Tracking Universal Features Through Fine-Tuning and Model Merging(https://arxiv.org/abs/2410.12391)
• Keywords: transformer
• Abstract: We study how features emerge, disappear, and persist across models fine-tuned on different domains of text. More specifically, we start from a base one-layer Transformer language model that is trained on a combination of the BabyLM corpus, and a collection of Python code from The Stack. This base model is adapted to two new domains of text: TinyStories, and the Lua programming language, respectively; and then these two models are merged using these two models using spherical linear interpolation. Our exploration aims to provide deeper insights into the stability and transformation of features across typical transfer-learning scenarios using small-scale models and sparse auto-encoders.

Title: Feature Augmentation for Self-supervised Contrastive Learning: A Closer Look

• Authors: Yong Zhang, Rui Zhu, Shifeng Zhang, Xu Zhou, Shifeng Chen, Xiaofan Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12396
• Pdf URL: https://arxiv.org/pdf/2410.12396
• Copy Paste: [[2410.12396]] Feature Augmentation for Self-supervised Contrastive Learning: A Closer Look(https://arxiv.org/abs/2410.12396)
• Keywords: robust
• Abstract: Self-supervised contrastive learning heavily relies on the view variance brought by data augmentation, so that it can learn a view-invariant pre-trained representation. Beyond increasing the view variance for contrast, this work focuses on improving the diversity of training data, to improve the generalization and robustness of the pre-trained models. To this end, we propose a unified framework to conduct data augmentation in the feature space, known as feature augmentation. This strategy is domain-agnostic, which augments similar features to the original ones and thus improves the data diversity. We perform a systematic investigation of various feature augmentation architectures, the gradient-flow skill, and the relationship between feature augmentation and traditional data augmentation. Our study reveals some practical principles for feature augmentation in self-contrastive learning. By integrating feature augmentation on the instance discrimination or the instance similarity paradigm, we consistently improve the performance of pre-trained feature learning and gain better generalization over the downstream image classification and object detection task.

Title: ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs

• Authors: Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12405
• Pdf URL: https://arxiv.org/pdf/2410.12405
• Copy Paste: [[2410.12405]] ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs(https://arxiv.org/abs/2410.12405)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but their performance is highly sensitive to the prompts utilized. This variability poses challenges for accurate assessment and user satisfaction. Current research frequently overlooks instance-level prompt variations and their implications on subjective evaluations. To address these shortcomings, we introduce ProSA, a framework designed to evaluate and comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity metric, PromptSensiScore, and leverages decoding confidence to elucidate underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers that prompt sensitivity fluctuates across datasets and models, with larger models exhibiting enhanced robustness. We observe that few-shot examples can alleviate this sensitivity issue, and subjective evaluations are also susceptible to prompt sensitivities, particularly in complex, reasoning-oriented tasks. Furthermore, our findings indicate that higher model confidence correlates with increased prompt robustness. We believe this work will serve as a helpful tool in studying prompt sensitivity of LLMs. The project is released at: this https URL .

Title: Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding

• Authors: Daichi Hayakawa, Issei Sato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12413
• Pdf URL: https://arxiv.org/pdf/2410.12413
• Copy Paste: [[2410.12413]] Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding(https://arxiv.org/abs/2410.12413)
• Keywords: transformer
• Abstract: In this study, we provide constructive proof that Transformers can recognize and generate hierarchical language efficiently with respect to model size, even without the need for a specific positional encoding. Specifically, we show that causal masking and a starting token enable Transformers to compute positional information and depth within hierarchical structures. We demonstrate that Transformers without positional encoding can generate hierarchical languages. Furthermore, we suggest that explicit positional encoding might have a detrimental effect on generalization with respect to sequence length.

Title: Adding web pentesting functionality to PTHelper

• Authors: María Olivares-Naya, Jacobo Casado de Gracia, Alfonso Sánchez-Macián
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.12422
• Pdf URL: https://arxiv.org/pdf/2410.12422
• Copy Paste: [[2410.12422]] Adding web pentesting functionality to PTHelper(https://arxiv.org/abs/2410.12422)
• Keywords: security, protect, attack
• Abstract: Web application pentesting is a crucial component in the offensive cybersecurity area, whose aim is to safeguard web applications and web services as the majority of the web applications are mounted in publicly accessible web environments. This method requires that the cybersecurity experts pretend and act as real attackers to identify all the errors and vulnerabilities in web applications with the objective of preventing and reducing damages. As this process may be quite complex and the amount of information pentesters need may be big, being able to automate it will help them to easily discover the vulnerabilities given. This project is the direct continuation of the previous initiative called PThelper: An open source tool to support the Penetration Testing process. This continuation is focused on expanding PThelper with the functionality to detect and later report web vulnerabilities in order to address emerging threats and strengthen the ability of the organizations to protect their web applications against potential cyber-attacks.

Title: Perseus: Leveraging Common Data Patterns with Curriculum Learning for More Robust Graph Neural Networks

• Authors: Kaiwen Xia, Huijun Wu, Duanyu Li, Min Xie, Ruibo Wang, Wenzhe Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12425
• Pdf URL: https://arxiv.org/pdf/2410.12425
• Copy Paste: [[2410.12425]] Perseus: Leveraging Common Data Patterns with Curriculum Learning for More Robust Graph Neural Networks(https://arxiv.org/abs/2410.12425)
• Keywords: defense, attack, robust
• Abstract: Graph Neural Networks (GNNs) excel at handling graph data but remain vulnerable to adversarial attacks. Existing defense methods typically rely on assumptions like graph sparsity and homophily to either preprocess the graph or guide structure learning. However, preprocessing methods often struggle to accurately distinguish between normal edges and adversarial perturbations, leading to suboptimal results due to the loss of valuable edge information. Robust graph neural network models train directly on graph data affected by adversarial perturbations, without preprocessing. This can cause the model to get stuck in poor local optima, negatively affecting its performance. To address these challenges, we propose Perseus, a novel adversarial defense method based on curriculum learning. Perseus assesses edge difficulty using global homophily and applies a curriculum learning strategy to adjust the learning order, guiding the model to learn the full graph structure while adaptively focusing on common data patterns. This approach mitigates the impact of adversarial perturbations. Experiments show that models trained with Perseus achieve superior performance and are significantly more robust to adversarial attacks.

Title: Conformity in Large Language Models

• Authors: Xiaochen Zhu, Caiqi Zhang, Tom Stafford, Nigel Collier, Andreas Vlachos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12428
• Pdf URL: https://arxiv.org/pdf/2410.12428
• Copy Paste: [[2410.12428]] Conformity in Large Language Models(https://arxiv.org/abs/2410.12428)
• Keywords: robust, large language model
• Abstract: The conformity effect describes the tendency of individuals to align their responses with the majority. Studying this bias in large language models (LLMs) is crucial, as LLMs are increasingly used in various information-seeking and decision-making tasks as conversation partners to improve productivity. Thus, conformity to incorrect responses can compromise their effectiveness. In this paper, we adapt psychological experiments to examine the extent of conformity in state-of-the-art LLMs. Our findings reveal that all models tested exhibit varying levels of conformity toward the majority, regardless of their initial choice or correctness, across different knowledge domains. Notably, we are the first to show that LLMs are more likely to conform when they are more uncertain in their own prediction. We further explore factors that influence conformity, such as training paradigms and input characteristics, finding that instruction-tuned models are less susceptible to conformity, while increasing the naturalness of majority tones amplifies conformity. Finally, we propose two interventions--Devil's Advocate and Question Distillation--to mitigate conformity, providing insights into building more robust language models.

Title: Reconstruction of Differentially Private Text Sanitization via Large Language Models

• Authors: Shuchao Pang, Zhigang Lu, Haichen Wang, Peng Fu, Yongbin Zhou, Minhui Xue, Bo Li
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12443
• Pdf URL: https://arxiv.org/pdf/2410.12443
• Copy Paste: [[2410.12443]] Reconstruction of Differentially Private Text Sanitization via Large Language Models(https://arxiv.org/abs/2410.12443)
• Keywords: security, privacy, attack, large language model
• Abstract: Differential privacy (DP) is the de facto privacy standard against privacy leakage attacks, including many recently discovered ones against large language models (LLMs). However, we discovered that LLMs could reconstruct the altered/removed privacy from given DP-sanitized prompts. We propose two attacks (black-box and white-box) based on the accessibility to LLMs and show that LLMs could connect the pair of DP-sanitized text and the corresponding private training data of LLMs by giving sample text pairs as instructions (in the black-box attacks) or fine-tuning data (in the white-box attacks). To illustrate our findings, we conduct comprehensive experiments on modern LLMs (e.g., LLaMA-2, LLaMA-3, ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, Claude-3, Claude-3.5, OPT, GPT-Neo, GPT-J, Gemma-2, and Pythia) using commonly used datasets (such as WikiMIA, Pile-CC, and Pile-Wiki) against both word-level and sentence-level DP. The experimental results show promising recovery rates, e.g., the black-box attacks against the word-level DP over WikiMIA dataset gave 72.18% on LLaMA-2 (70B), 82.39% on LLaMA-3 (70B), 75.35% on Gemma-2, 91.2% on ChatGPT-4o, and 94.01% on Claude-3.5 (Sonnet). More urgently, this study indicates that these well-known LLMs have emerged as a new security risk for existing DP text sanitization approaches in the current environment.

Title: Expanding Chatbot Knowledge in Customer Service: Context-Aware Similar Question Generation Using Large Language Models

• Authors: Mengze Hong, Yuanfeng Song, Di Jiang, Lu Wang, Zichang Guo, Chen Jason Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12444
• Pdf URL: https://arxiv.org/pdf/2410.12444
• Copy Paste: [[2410.12444]] Expanding Chatbot Knowledge in Customer Service: Context-Aware Similar Question Generation Using Large Language Models(https://arxiv.org/abs/2410.12444)
• Keywords: large language model
• Abstract: Reliable responses of service chatbots are often achieved by employing retrieval-based methods that restrict answers to a knowledge base comprising predefined question-answer pairs (QA pairs). To accommodate potential variations in how a customer's query may be expressed, it emerges as the favored solution to augment these QA pairs with similar questions that are possibly diverse while remaining semantic consistency. This augmentation task is known as Similar Question Generation (SQG). Traditional methods that heavily rely on human efforts or rule-based techniques suffer from limited diversity or significant semantic deviation from the source question, only capable of producing a finite number of useful questions. To address these limitations, we propose an SQG approach based on Large Language Models (LLMs), capable of producing a substantial number of diverse questions while maintaining semantic consistency to the source QA pair. This is achieved by leveraging LLMs' natural language understanding capability through fine-tuning with specially designed prompts. The experiments conducted on a real customer-service dataset demonstrate that our method surpasses baseline methods by a significant margin in terms of semantic diversity. Human evaluation further confirms that integrating the answer that reflects the customer's intention is crucial for increasing the number of generated questions that meet business requirements.

Title: Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMs

• Authors: Hyeonwoo Kim, Dahyun Kim, Jihoo Kim, Sukyung Lee, Yungi Kim, Chanjun Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12445
• Pdf URL: https://arxiv.org/pdf/2410.12445
• Copy Paste: [[2410.12445]] Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMs(https://arxiv.org/abs/2410.12445)
• Keywords: large language model
• Abstract: The Open Ko-LLM Leaderboard has been instrumental in benchmarking Korean Large Language Models (LLMs), yet it has certain limitations. Notably, the disconnect between quantitative improvements on the overly academic leaderboard benchmarks and the qualitative impact of the models should be addressed. Furthermore, the benchmark suite is largely composed of translated versions of their English counterparts, which may not fully capture the intricacies of the Korean language. To address these issues, we propose Open Ko-LLM Leaderboard2, an improved version of the earlier Open Ko-LLM Leaderboard. The original benchmarks are entirely replaced with new tasks that are more closely aligned with real-world capabilities. Additionally, four new native Korean benchmarks are introduced to better reflect the distinct characteristics of the Korean language. Through these refinements, Open Ko-LLM Leaderboard2 seeks to provide a more meaningful evaluation for advancing Korean LLMs.

Title: FairGLVQ: Fairness in Partition-Based Classification

• Authors: Felix Störck, Fabian Hinder, Johannes Brinkrolf, Benjamin Paassen, Valerie Vaquet, Barbara Hammer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12452
• Pdf URL: https://arxiv.org/pdf/2410.12452
• Copy Paste: [[2410.12452]] FairGLVQ: Fairness in Partition-Based Classification(https://arxiv.org/abs/2410.12452)
• Keywords: security, fair
• Abstract: Fairness is an important objective throughout society. From the distribution of limited goods such as education, over hiring and payment, to taxes, legislation, and jurisprudence. Due to the increasing importance of machine learning approaches in all areas of daily life including those related to health, security, and equity, an increasing amount of research focuses on fair machine learning. In this work, we focus on the fairness of partition- and prototype-based models. The contribution of this work is twofold: 1) we develop a general framework for fair machine learning of partition-based models that does not depend on a specific fairness definition, and 2) we derive a fair version of learning vector quantization (LVQ) as a specific instantiation. We compare the resulting algorithm against other algorithms from the literature on theoretical and real-world data showing its practical relevance.

Title: Training Neural Samplers with Reverse Diffusive KL Divergence

• Authors: Jiajun He, Wenlin Chen, Mingtian Zhang, David Barber, José Miguel Hernández-Lobato
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12456
• Pdf URL: https://arxiv.org/pdf/2410.12456
• Copy Paste: [[2410.12456]] Training Neural Samplers with Reverse Diffusive KL Divergence(https://arxiv.org/abs/2410.12456)
• Keywords: diffusion, generative
• Abstract: Training generative models to sample from unnormalized density functions is an important and challenging task in machine learning. Traditional training methods often rely on the reverse Kullback-Leibler (KL) divergence due to its tractability. However, the mode-seeking behavior of reverse KL hinders effective approximation of multi-modal target distributions. To address this, we propose to minimize the reverse KL along diffusion trajectories of both model and target densities. We refer to this objective as the reverse diffusive KL divergence, which allows the model to capture multiple modes. Leveraging this objective, we train neural samplers that can efficiently generate samples from the target distribution in one step. We demonstrate that our method enhances sampling performance across various Boltzmann distributions, including both synthetic multi-modal densities and n-body particle systems.

Title: The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph

• Authors: Minghao Wu, Thuy-Trang Vu, Lizhen Qu, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12458
• Pdf URL: https://arxiv.org/pdf/2410.12458
• Copy Paste: [[2410.12458]] The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph(https://arxiv.org/abs/2410.12458)
• Keywords: large language model
• Abstract: The performance of large language models (LLMs) in natural language processing (NLP) tasks is significantly influenced by the quality and diversity of data used for supervised fine-tuning (SFT). Current data selection methods often focus solely on quality or diversity, leading to underperforming models due to suboptimal training data. In this paper, we introduce GraphFilter, a novel method that represents the dataset as a bipartite graph, linking sentences to their constituent n-grams. This representation effectively captures the relationships between sentences and linguistic patterns, facilitating the selection of sentences that enhance n-gram diversity. To balance quality and diversity during selection, we propose a priority function that combines the quality metric with the diversity metric in a multiplicative manner. GraphFilter iteratively selects high-priority sentences, updates the bipartite graph by removing covered n-grams, and re-calculates priorities to reflect the evolving data landscape. We conduct extensive experiments using three model backbones across six widely used benchmarks. The results demonstrate that GraphFilter outperforms all nine baseline approaches, achieving superior model performance and computational efficiency. Our analyses validate the effectiveness of our design choices, examine the subsets selected by GraphFilter and other methods, highlight the importance of instruction diversity, and explore the role of quality and diversity in relation to subset sizes. GraphFilter establishes a new foundation for effective data selection strategies, encouraging further research in data selection for LLMs.

Title: HELM: Hierarchical Encoding for mRNA Language Modeling

• Authors: Mehdi Yazdani-Jahromi, Mangal Prakash, Tommaso Mansi, Artem Moskalev, Rui Liao
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2410.12459
• Pdf URL: https://arxiv.org/pdf/2410.12459
• Copy Paste: [[2410.12459]] HELM: Hierarchical Encoding for mRNA Language Modeling(https://arxiv.org/abs/2410.12459)
• Keywords: generative
• Abstract: Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on six diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8\%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines.

Title: Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention

• Authors: Weixuan Wang, Minghao Wu, Barry Haddow, Alexandra Birch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12462
• Pdf URL: https://arxiv.org/pdf/2410.12462
• Copy Paste: [[2410.12462]] Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention(https://arxiv.org/abs/2410.12462)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities in natural language processing but exhibit significant performance gaps among different languages. Most existing approaches to address these disparities rely on pretraining or fine-tuning, which are resource-intensive. To overcome these limitations without incurring significant costs, we propose Inference-Time Cross-Lingual Intervention (INCLINE), a novel framework that enhances LLM performance on low-performing (source) languages by aligning their internal representations with those of high-performing (target) languages during inference. INCLINE initially learns alignment matrices using parallel sentences from source and target languages through a Least-Squares optimization, and then applies these matrices during inference to transform the low-performing language representations toward the high-performing language space. Extensive experiments on nine benchmarks with five LLMs demonstrate that INCLINE significantly improves performance across diverse tasks and languages, compared to recent strong baselines. Our analysis demonstrates that INCLINE is highly cost-effective and applicable to a wide range of applications. In addition, we release the code to foster research along this line: this https URL.

Title: RADS-Checker: Measuring Compliance with Right of Access by the Data Subject in Android Markets

• Authors: Zhenhua Li, Zhanpeng Liang, Congcong Yao, Jingyu Hua, Sheng Zhong
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.12463
• Pdf URL: https://arxiv.org/pdf/2410.12463
• Copy Paste: [[2410.12463]] RADS-Checker: Measuring Compliance with Right of Access by the Data Subject in Android Markets(https://arxiv.org/abs/2410.12463)
• Keywords: privacy, protect
• Abstract: The latest data protection regulations worldwide, such as the General Data Protection Regulation (GDPR), have established the Right of Access by the Data Subject (RADS), granting users the right to access and obtain a copy of their personal data from the data controllers. This clause can effectively compel data controllers to handle user personal data more cautiously, which is of significant importance for protecting user privacy. However, there is currently no research systematically examining whether RADS has been effectively implemented in mobile apps, which are the most common personal data controllers. In this study, we propose a compliance measurement framework for RADS in apps. In our framework, we first analyze an app's privacy policy text using NLP techniques such as GPT-4 to verify whether it clearly declares offering RADS to users and provides specific details on how the right can be exercised. Next, we assess the authenticity and usability of the identified implementation methods by submitting data access requests to the app. Finally, for the obtained data copies, we further verify their completeness by comparing them with the user personal data actually collected by the app during runtime, as captured by Frida Hook. We analyzed a total of 1,631 apps in the American app market G and the Chinese app market H. The results show that less than 54.50% and 37.05% of apps in G and H, respectively, explicitly state in their privacy policies that they can provide users with copies of their personal data. Additionally, in both app markets, less than 20% of apps could truly provide users with their data copies. Finally, among the obtained data copies, only about 2.94% from G pass the completeness verification.

Title: Learning to Predict Usage Options of Product Reviews with LLM-Generated Labels

• Authors: Leo Kohlenberg, Leonard Horns, Frederic Sadrieh, Nils Kiele, Matthis Clausen, Konstantin Ketterer, Avetis Navasardyan, Tamara Czinczoll, Gerard de Melo, Ralf Herbrich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12470
• Pdf URL: https://arxiv.org/pdf/2410.12470
• Copy Paste: [[2410.12470]] Learning to Predict Usage Options of Product Reviews with LLM-Generated Labels(https://arxiv.org/abs/2410.12470)
• Keywords: privacy
• Abstract: Annotating large datasets can be challenging. However, crowd-sourcing is often expensive and can lack quality, especially for non-trivial tasks. We propose a method of using LLMs as few-shot learners for annotating data in a complex natural language task where we learn a standalone model to predict usage options for products from customer reviews. We also propose a new evaluation metric for this scenario, HAMS4, that can be used to compare a set of strings with multiple reference sets. Learning a custom model offers individual control over energy efficiency and privacy measures compared to using the LLM directly for the sequence-to-sequence task. We compare this data annotation approach with other traditional methods and demonstrate how LLMs can enable considerable cost savings. We find that the quality of the resulting data exceeds the level attained by third-party vendor services and that GPT-4-generated labels even reach the level of domain experts. We make the code and generated labels publicly available.

Title: Retrieval-Reasoning Large Language Model-based Synthetic Clinical Trial Generation

• Authors: Zerui Xu, Fang Wu, Tianfan Fu, Yue Zhao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12476
• Pdf URL: https://arxiv.org/pdf/2410.12476
• Copy Paste: [[2410.12476]] Retrieval-Reasoning Large Language Model-based Synthetic Clinical Trial Generation(https://arxiv.org/abs/2410.12476)
• Keywords: privacy, large language model
• Abstract: Machine learning (ML) exhibits promise in the clinical domain. However, it is constrained by data scarcity and ethical considerations, as the generation of clinical trials presents significant challenges due to stringent privacy regulations, high costs, and the extended duration required for conducting studies with human participants. Despite the advancements of large language models (LLMs) in general generation tasks, their potential in facilitating the generation of synthetic clinical trials is under-explored. To address this gap, we introduce a novel Retrieval-Reasoning few-shot framework that leverages LLMs to generate artificial yet realistic and diverse clinical trials with binary success/failure labels. Experiments conducted on real clinical trials from the \url{this http URL} database demonstrate that our synthetic data can effectively augment real datasets. Furthermore, by fine-tuning a pre-trained model as a binary classifier on synthetic clinical trial datasets, we demonstrate that this augmentation enhances model training for downstream tasks such as trial outcome prediction. Our findings suggest that LLMs for synthetic clinical trial generation hold promise for accelerating clinical research and upholding ethical standards for patient privacy. The code is publicly available at this https URL.

Title: MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

• Authors: Boyang Xue, Hongru Wang, Rui Wang, Sheng Wang, Zezhong Wang, Yiming Du, Bin Liang, Kam-Fai Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12478
• Pdf URL: https://arxiv.org/pdf/2410.12478
• Copy Paste: [[2410.12478]] MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models(https://arxiv.org/abs/2410.12478)
• Keywords: large language model
• Abstract: The tendency of Large Language Models (LLMs) to generate hallucinations raises concerns regarding their reliability. Therefore, confidence estimations indicating the extent of trustworthiness of the generations become essential. However, current LLM confidence estimations in languages other than English remain underexplored. This paper addresses this gap by introducing a comprehensive investigation of Multilingual Confidence estimation (MlingConf) on LLMs, focusing on both language-agnostic (LA) and language-specific (LS) tasks to explore the performance and language dominance effects of multilingual confidence estimations on different tasks. The benchmark comprises four meticulously checked and human-evaluate high-quality multilingual datasets for LA tasks and one for the LS task tailored to specific social, cultural, and geographical contexts of a language. Our experiments reveal that on LA tasks English exhibits notable linguistic dominance in confidence estimations than other languages, while on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance in multilingual confidence estimations. The phenomena inspire a simple yet effective native-tone prompting strategy by employing language-specific prompts for LS tasks, effectively improving LLMs' reliability and accuracy on LS tasks.

Title: KcMF: A Knowledge-compliant Framework for Schema and Entity Matching with Fine-tuning-free LLMs

• Authors: Yongqin Xu, Huan Li, Ke Chen, Lidan Shou
• Subjects: cs.CL, cs.AI, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12480
• Pdf URL: https://arxiv.org/pdf/2410.12480
• Copy Paste: [[2410.12480]] KcMF: A Knowledge-compliant Framework for Schema and Entity Matching with Fine-tuning-free LLMs(https://arxiv.org/abs/2410.12480)
• Keywords: large language model
• Abstract: Schema and entity matching tasks are crucial for data integration and management. While large language models (LLMs) have shown promising results in these tasks, they suffer from hallucinations and confusion about task instructions. In this paper, we present the Knowledge-Compliant Matching Framework (KcMF), an LLM-based approach that addresses these issues without the need for domain-specific fine-tuning. KcMF employs a pseudo-code-based task decomposition strategy to adopt task-specific natural language statements that guide LLM reasoning and reduce confusion. We also propose two mechanisms, Dataset as Knowledge (DaK) and Example as Knowledge (EaK), to build domain knowledge sets when unstructured domain knowledge is lacking. Additionally, we introduce a result-ensembling strategy to leverage multiple knowledge sources and suppress poorly formatted outputs. Comprehensive evaluations on schema and entity matching tasks demonstrate that KcMF outperforms previous non-LLM state-of-the-art (SOTA) methods by an average F1 score of 22.9% and competes effectively with SOTA fine-tuned LLMs. Moreover, KcMF generalizes well across different LLMs.

Title: SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling

• Authors: Loris Gaven, Clement Romac, Thomas Carta, Sylvain Lamprier, Olivier Sigaud, Pierre-Yves Oudeyer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12481
• Pdf URL: https://arxiv.org/pdf/2410.12481
• Copy Paste: [[2410.12481]] SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling(https://arxiv.org/abs/2410.12481)
• Keywords: generative, large language model
• Abstract: The past years have seen Large Language Models (LLMs) strive not only as generative models but also as agents solving textual sequential decision-making tasks. When facing complex environments where their zero-shot abilities are insufficient, recent work showed online Reinforcement Learning (RL) could be used for the LLM agent to discover and learn efficient strategies interactively. However, most prior work sticks to on-policy algorithms, which greatly reduces the scope of methods such agents could use for both exploration and exploitation, such as experience replay and hindsight relabeling. Yet, such methods may be key for LLM learning agents, and in particular when designing autonomous intrinsically motivated agents sampling and pursuing their own goals (i.e. autotelic agents). This paper presents and studies an adaptation of Soft Actor-Critic and hindsight relabeling to LLM agents. Our method not only paves the path towards autotelic LLM agents that learn online but can also outperform on-policy methods in more classic multi-goal RL environments.

Title: Synthetic Augmentation for Anatomical Landmark Localization using DDPMs

• Authors: Arnela Hadzic, Lea Bogensperger, Simon Johannes Joham, Martin Urschler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12489
• Pdf URL: https://arxiv.org/pdf/2410.12489
• Copy Paste: [[2410.12489]] Synthetic Augmentation for Anatomical Landmark Localization using DDPMs(https://arxiv.org/abs/2410.12489)
• Keywords: diffusion, generative
• Abstract: Deep learning techniques for anatomical landmark localization (ALL) have shown great success, but their reliance on large annotated datasets remains a problem due to the tedious and costly nature of medical data acquisition and annotation. While traditional data augmentation, variational autoencoders (VAEs), and generative adversarial networks (GANs) have already been used to synthetically expand medical datasets, diffusion-based generative models have recently started to gain attention for their ability to generate high-quality synthetic images. In this study, we explore the use of denoising diffusion probabilistic models (DDPMs) for generating medical images and their corresponding heatmaps of landmarks to enhance the training of a supervised deep learning model for ALL. Our novel approach involves a DDPM with a 2-channel input, incorporating both the original medical image and its heatmap of annotated landmarks. We also propose a novel way to assess the quality of the generated images using a Markov Random Field (MRF) model for landmark matching and a Statistical Shape Model (SSM) to check landmark plausibility, before we evaluate the DDPM-augmented dataset in the context of an ALL task involving hand X-Rays.

Title: Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective

• Authors: Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, Lidong Bing
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12490
• Pdf URL: https://arxiv.org/pdf/2410.12490
• Copy Paste: [[2410.12490]] Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective(https://arxiv.org/abs/2410.12490)
• Keywords: diffusion, generative
• Abstract: Latent-based image generative models, such as Latent Diffusion Models (LDMs) and Mask Image Models (MIMs), have achieved notable success in image generation tasks. These models typically leverage reconstructive autoencoders like VQGAN or VAE to encode pixels into a more compact latent space and learn the data distribution in the latent space instead of directly from pixels. However, this practice raises a pertinent question: Is it truly the optimal choice? In response, we begin with an intriguing observation: despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image generation. This finding contrasts sharply with the field of NLP, where the autoregressive model GPT has established a commanding presence. To address this discrepancy, we introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Furthermore, we propose a simple but effective discrete image tokenizer to stabilize the latent space for image generative modeling. Experimental results show that image autoregressive modeling with our tokenizer (DiGIT) benefits both image understanding and image generation with the next token prediction principle, which is inherently straightforward for GPT models but challenging for other generative models. Remarkably, for the first time, a GPT-style autoregressive model for images outperforms LDMs, which also exhibits substantial improvement akin to GPT when scaling up model size. Our findings underscore the potential of an optimized latent space and the integration of discrete tokenization in advancing the capabilities of image generative models. The code is available at \url{this https URL}.

Title: Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse RL

• Authors: Jared Joselowitz, Arjun Jagota, Satyapriya Krishna, Sonali Parbhoo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12491
• Pdf URL: https://arxiv.org/pdf/2410.12491
• Copy Paste: [[2410.12491]] Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse RL(https://arxiv.org/abs/2410.12491)
• Keywords: interpretability, large language model
• Abstract: Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 80.40% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems.

Title: With a Grain of SALT: Are LLMs Fair Across Social Dimensions?

• Authors: Samee Arif, Zohaib Khan, Agha Ali Raza, Awais Athar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12499
• Pdf URL: https://arxiv.org/pdf/2410.12499
• Copy Paste: [[2410.12499]] With a Grain of SALT: Are LLMs Fair Across Social Dimensions?(https://arxiv.org/abs/2410.12499)
• Keywords: fair, large language model
• Abstract: This paper presents an analysis of biases in open-source Large Language Models (LLMs) across various genders, religions, and races. We introduce a methodology for generating a bias detection dataset using seven bias triggers: General Debate, Positioned Debate, Career Advice, Story Generation, Problem-Solving, Cover-Letter Writing, and CV Generation. We use GPT-4o to generate a diverse set of prompts for each trigger across various genders, religious and racial groups. We evaluate models from Llama and Gemma family on the generated dataset. We anonymise the LLM-generated text associated with each group using GPT-4o-mini and do a pairwise comparison using GPT-4o-as-a-Judge. To quantify bias in the LLM-generated text we use the number of wins and losses in the pairwise comparison. Our analysis spans three languages, English, German, and Arabic to explore how language influences bias manifestation. Our findings reveal that LLMs exhibit strong polarization toward certain groups across each category, with a notable consistency observed across models. However, when switching languages, variations and anomalies emerge, often attributable to cultural cues and contextual differences.

Title: DH-VTON: Deep Text-Driven Virtual Try-On via Hybrid Attention Learning

• Authors: Jiabao Wei, Zhiyuan Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12501
• Pdf URL: https://arxiv.org/pdf/2410.12501
• Copy Paste: [[2410.12501]] DH-VTON: Deep Text-Driven Virtual Try-On via Hybrid Attention Learning(https://arxiv.org/abs/2410.12501)
• Keywords: extraction, diffusion
• Abstract: Virtual Try-ON (VTON) aims to synthesis specific person images dressed in given garments, which recently receives numerous attention in online shopping scenarios. Currently, the core challenges of the VTON task mainly lie in the fine-grained semantic extraction (i.e.,deep semantics) of the given reference garments during depth estimation and effective texture preservation when the garments are synthesized and warped onto human body. To cope with these issues, we propose DH-VTON, a deep text-driven virtual try-on model featuring a special hybrid attention learning strategy and deep garment semantic preservation module. By standing on the shoulder of a well-built pre-trained paint-by-example (abbr. PBE) approach, we present our DH-VTON pipeline in this work. Specifically, to extract the deep semantics of the garments, we first introduce InternViT-6B as fine-grained feature learner, which can be trained to align with the large-scale intrinsic knowledge with deep text semantics (e.g.,"neckline" or "girdle") to make up for the deficiency of the commonly adopted CLIP encoder. Based on this, to enhance the customized dressing abilities, we further introduce Garment-Feature ControlNet Plus (abbr. GFC+) module and propose to leverage a fresh hybrid attention strategy for training, which can adaptively integrate fine-grained characteristics of the garments into the different layers of the VTON model, so as to achieve multi-scale features preservation effects. Extensive experiments on several representative datasets demonstrate that our method outperforms previous diffusion-based and GAN-based approaches, showing competitive performance in preserving garment details and generating authentic human images.

Title: Advancing Fairness in Natural Language Processing: From Traditional Methods to Explainability

• Authors: Fanny Jourdan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12511
• Pdf URL: https://arxiv.org/pdf/2410.12511
• Copy Paste: [[2410.12511]] Advancing Fairness in Natural Language Processing: From Traditional Methods to Explainability(https://arxiv.org/abs/2410.12511)
• Keywords: fair, explainability, transformer
• Abstract: The burgeoning field of Natural Language Processing (NLP) stands at a critical juncture where the integration of fairness within its frameworks has become an imperative. This PhD thesis addresses the need for equity and transparency in NLP systems, recognizing that fairness in NLP is not merely a technical challenge but a moral and ethical necessity, requiring a rigorous examination of how these technologies interact with and impact diverse human populations. Through this lens, this thesis undertakes a thorough investigation into the development of equitable NLP methodologies and the evaluation of biases that prevail in current systems. First, it introduces an innovative algorithm to mitigate biases in multi-class classifiers, tailored for high-risk NLP applications, surpassing traditional methods in both bias mitigation and prediction accuracy. Then, an analysis of the Bios dataset reveals the impact of dataset size on discriminatory biases and the limitations of standard fairness metrics. This awareness has led to explorations in the field of explainable AI, aiming for a more complete understanding of biases where traditional metrics are limited. Consequently, the thesis presents COCKATIEL, a model-agnostic explainability method that identifies and ranks concepts in Transformer models, outperforming previous approaches in sentiment analysis tasks. Finally, the thesis contributes to bridging the gap between fairness and explainability by introducing TaCo, a novel method to neutralize bias in Transformer model embeddings. In conclusion, this thesis constitutes a significant interdisciplinary endeavor that intertwines explicability and fairness to challenge and reshape current NLP paradigms. The methodologies and critiques presented contribute to the ongoing discourse on fairness in machine learning, offering actionable solutions for more equitable and responsible AI systems.

Title: FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction

• Authors: Akriti Jain, Saransh Sharma, Koyel Mukherjee, Soumyabrata Pal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12513
• Pdf URL: https://arxiv.org/pdf/2410.12513
• Copy Paste: [[2410.12513]] FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction(https://arxiv.org/abs/2410.12513)
• Keywords: transformer, large language model
• Abstract: Auto-regressive Large Language Models (LLMs) demonstrate remarkable performance across domanins such as vision and language processing. However, due to sequential processing through a stack of transformer layers, autoregressive decoding faces significant computation/latency challenges, particularly in resource constrained environments like mobile and edge devices. Existing approaches in literature that aim to improve latency via skipping layers have two distinct flavors - 1) Early exit 2) Input-agnostic heuristics where tokens exit at pre-determined layers irrespective of input sequence. Both the above strategies have limitations - the former cannot be applied to handle KV Caching necessary for speed-ups in modern framework and the latter does not capture the variation in layer importance across tasks or more generally, across input sequences. To address both limitations, we propose FIRST, an algorithm that reduces inference latency by using layer-specific routers to select a subset of transformer layers adaptively for each input sequence - the prompt (during prefill stage) decides which layers will be skipped during decoding. FIRST preserves compatibility with KV caching enabling faster inference while being quality-aware. FIRST is model-agnostic and can be easily enabled on any pre-trained LLM. We further improve performance by incorporating LoRA adapters for fine-tuning on external datasets, enhancing task-specific accuracy while maintaining latency benefits. Our approach reveals that input adaptivity is critical - indeed, different task-specific middle layers play a crucial role in evolving hidden representations depending on task. Extensive experiments show that FIRST significantly reduces latency while retaining competitive performance (as compared to baselines), making our approach an efficient solution for LLM deployment in low-resource environments.

Title: QueensCAMP: an RGB-D dataset for robust Visual SLAM

• Authors: Hudson M. S. Bruno, Esther L. Colombini, Sidney N. Givigi Jr
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12520
• Pdf URL: https://arxiv.org/pdf/2410.12520
• Copy Paste: [[2410.12520]] QueensCAMP: an RGB-D dataset for robust Visual SLAM(https://arxiv.org/abs/2410.12520)
• Keywords: robust
• Abstract: Visual Simultaneous Localization and Mapping (VSLAM) is a fundamental technology for robotics applications. While VSLAM research has achieved significant advancements, its robustness under challenging situations, such as poor lighting, dynamic environments, motion blur, and sensor failures, remains a challenging issue. To address these challenges, we introduce a novel RGB-D dataset designed for evaluating the robustness of VSLAM systems. The dataset comprises real-world indoor scenes with dynamic objects, motion blur, and varying illumination, as well as emulated camera failures, including lens dirt, condensation, underexposure, and overexposure. Additionally, we offer open-source scripts for injecting camera failures into any images, enabling further customization by the research community. Our experiments demonstrate that ORB-SLAM2, a traditional VSLAM algorithm, and TartanVO, a Deep Learning-based VO algorithm, can experience performance degradation under these challenging conditions. Therefore, this dataset and the camera failure open-source tools provide a valuable resource for developing more robust VSLAM systems capable of handling real-world challenges.

Title: MING: A Functional Approach to Learning Molecular Generative Models

• Authors: Van Khoa Nguyen, Maciej Falkiewicz, Giangiacomo Mercatali, Alexandros Kalousis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12522
• Pdf URL: https://arxiv.org/pdf/2410.12522
• Copy Paste: [[2410.12522]] MING: A Functional Approach to Learning Molecular Generative Models(https://arxiv.org/abs/2410.12522)
• Keywords: diffusion, generative
• Abstract: Traditional molecule generation methods often rely on sequence or graph-based representations, which can limit their expressive power or require complex permutation-equivariant architectures. This paper introduces a novel paradigm for learning molecule generative models based on functional representations. Specifically, we propose Molecular Implicit Neural Generation (MING), a diffusion-based model that learns molecular distributions in function space. Unlike standard diffusion processes in data space, MING employs a novel functional denoising probabilistic process, which jointly denoises the information in both the function's input and output spaces by leveraging an expectation-maximization procedure for latent implicit neural representations of data. This approach allows for a simple yet effective model design that accurately captures underlying function distributions. Experimental results on molecule-related datasets demonstrate MING's superior performance and ability to generate plausible molecular samples, surpassing state-of-the-art data-space methods while offering a more streamlined architecture and significantly faster generation times.

Title: Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing

• Authors: Mingce Guo, Jingxuan He, Shengeng Tang, Zhangye Wang, Lechao Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12526
• Pdf URL: https://arxiv.org/pdf/2410.12526
• Copy Paste: [[2410.12526]] Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing(https://arxiv.org/abs/2410.12526)
• Keywords: diffusion, generative
• Abstract: Text-driven video editing utilizing generative diffusion models has garnered significant attention due to their potential applications. However, existing approaches are constrained by the limited word embeddings provided in pre-training, which hinders nuanced editing targeting open concepts with specific attributes. Directly altering the keywords in target prompts often results in unintended disruptions to the attention mechanisms. To achieve more flexible editing easily, this work proposes an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly by devising abstract conceptual pairs. Specifically, the framework involves concept-augmented textual inversion and a dual prior supervision mechanism. The former enables plug-and-play guidance of stable diffusion for video editing, effectively capturing target attributes for more stylized results. The dual prior supervision mechanism significantly enhances video stability and fidelity. Comprehensive evaluations demonstrate that our approach generates more stable and lifelike videos, outperforming state-of-the-art methods.

Title: MedAide: Towards an Omni Medical Aide via Specialized LLM-based Multi-Agent Collaboration

• Authors: Jinjie Wei, Dingkang Yang, Yanshu Li, Qingyao Xu, Zhaoyu Chen, Mingcheng Li, Yue Jiang, Xiaolu Hou, Lihua Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12532
• Pdf URL: https://arxiv.org/pdf/2410.12532
• Copy Paste: [[2410.12532]] MedAide: Towards an Omni Medical Aide via Specialized LLM-based Multi-Agent Collaboration(https://arxiv.org/abs/2410.12532)
• Keywords: large language model
• Abstract: Large Language Model (LLM)-driven interactive systems currently show potential promise in healthcare domains. Despite their remarkable capabilities, LLMs typically lack personalized recommendations and diagnosis analysis in sophisticated medical applications, causing hallucinations and performance bottlenecks. To address these challenges, this paper proposes MedAide, an LLM-based omni medical multi-agent collaboration framework for specialized healthcare services. Specifically, MedAide first performs query rewriting through retrieval-augmented generation to accomplish accurate medical intent understanding. Immediately, we devise a contextual encoder to obtain intent prototype embeddings, which are used to recognize fine-grained intents by similarity matching. According to the intent relevance, the activated agents collaborate effectively to provide integrated decision analysis. Extensive experiments are conducted on four medical benchmarks with composite intents. Experimental results from automated metrics and expert doctor evaluations show that MedAide outperforms current LLMs and improves their medical proficiency and strategic reasoning.

Title: SEMSO: A Secure and Efficient Multi-Data Source Blockchain Oracle

• Authors: Youquan Xian, Xueying Zeng, Chunpei Li, Peng Wang, Dongcheng Li, Peng Liu, Xianxian Li
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.12540
• Pdf URL: https://arxiv.org/pdf/2410.12540
• Copy Paste: [[2410.12540]] SEMSO: A Secure and Efficient Multi-Data Source Blockchain Oracle(https://arxiv.org/abs/2410.12540)
• Keywords: secure, security
• Abstract: In recent years, blockchain oracle, as the key link between blockchain and real-world data interaction, has greatly expanded the application scope of blockchain. In particular, the emergence of the Multi-Data Source (MDS) oracle has greatly improved the reliability of the oracle in the case of untrustworthy data sources. However, the current MDS oracle scheme requires nodes to obtain data redundantly from multiple data sources to guarantee data reliability, which greatly increases the resource overhead and response time of the system. Therefore, in this paper, we propose a Secure and Efficient Multi-data Source Oracle framework (SEMSO), which nodes only need to access one data source to ensure the reliability of final data. First, we design a new off-chain data aggregation protocol TBLS, to guarantee data source diversity and reliability at low cost. Second, according to the rational man assumption, the data source selection task of nodes is modeled and solved based on the Bayesian game under incomplete information to maximize the node's revenue while improving the success rate of TBLS aggregation and system response speed. Security analysis verifies the reliability of the proposed scheme, and experiments show that under the same environmental assumptions, SEMSO takes into account data diversity while reducing the response time by 23.5\%.

Title: LLM-based Translation Inference with Iterative Bilingual Understanding

• Authors: Andong Chen, Kehai Chen, Yang Xiang, Xuefeng Bai, Muyun Yang, Tiejun Zhao, Min zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12543
• Pdf URL: https://arxiv.org/pdf/2410.12543
• Copy Paste: [[2410.12543]] LLM-based Translation Inference with Iterative Bilingual Understanding(https://arxiv.org/abs/2410.12543)
• Keywords: large language model
• Abstract: The remarkable understanding and generation capabilities of large language models (LLMs) have greatly improved translation performance. However, incorrect understanding of the sentence to be translated can degrade translation quality. To address this issue, we proposed a novel Iterative Bilingual Understanding Translation (IBUT) method based on the cross-lingual capabilities of LLMs and the dual characteristics of translation tasks. The cross-lingual capability of LLMs enables the generation of contextual understanding for both the source and target languages separately. Furthermore, the dual characteristics allow IBUT to generate effective cross-lingual feedback, iteratively refining contextual understanding, thereby reducing errors and improving translation performance. Experimental results showed that the proposed IBUT outperforms several strong comparison methods, especially being generalized to multiple domains (e.g., news, commonsense, and cultural translation benchmarks).

Title: One Step Diffusion via Shortcut Models

• Authors: Kevin Frans, Danijar Hafner, Sergey Levine, Pieter Abbeel
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12557
• Pdf URL: https://arxiv.org/pdf/2410.12557
• Copy Paste: [[2410.12557]] One Step Diffusion via Shortcut Models(https://arxiv.org/abs/2410.12557)
• Keywords: diffusion, generative
• Abstract: Diffusion models and flow-matching models have enabled generating diverse and realistic images by learning to transfer noise to data. However, sampling from these models involves iterative denoising over many neural network passes, making generation slow and expensive. Previous approaches for speeding up sampling require complex training regimes, such as multiple training phases, multiple networks, or fragile scheduling. We introduce shortcut models, a family of generative models that use a single network and training phase to produce high-quality samples in a single or multiple sampling steps. Shortcut models condition the network not only on the current noise level but also on the desired step size, allowing the model to skip ahead in the generation process. Across a wide range of sampling step budgets, shortcut models consistently produce higher quality samples than previous approaches, such as consistency models and reflow. Compared to distillation, shortcut models reduce complexity to a single network and training phase and additionally allow varying step budgets at inference time.

Title: Development of Image Collection Method Using YOLO and Siamese Network

• Authors: Chan Young Shin, Ah Hyun Lee, Jun Young Lee, Ji Min Lee, Soo Jin Park
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12561
• Pdf URL: https://arxiv.org/pdf/2410.12561
• Copy Paste: [[2410.12561]] Development of Image Collection Method Using YOLO and Siamese Network(https://arxiv.org/abs/2410.12561)
• Keywords: robust
• Abstract: As we enter the era of big data, collecting high-quality data is very important. However, collecting data by humans is not only very time-consuming but also expensive. Therefore, many scientists have devised various methods to collect data using computers. Among them, there is a method called web crawling, but the authors found that the crawling method has a problem in that unintended data is collected along with the user. The authors found that this can be filtered using the object recognition model YOLOv10. However, there are cases where data that is not properly filtered remains. Here, image reclassification was performed by additionally utilizing the distance output from the Siamese network, and higher performance was recorded than other classification models. (average \_f1 score YOLO+MobileNet 0.678->YOLO+SiameseNet 0.772)) The user can specify a distance threshold to adjust the balance between data deficiency and noise-robustness. The authors also found that the Siamese network can achieve higher performance with fewer resources because the cropped images are used for object recognition when processing images in the Siamese network. (Class 20 mean-based f1 score, non-crop+Siamese(MobileNetV3-Small) 80.94 -> crop preprocessing+Siamese(MobileNetV3-Small) 82.31) In this way, the image retrieval system that utilizes two consecutive models to reduce errors can save users' time and effort, and build better quality data faster and with fewer resources than before.

Title: Adaptive Prompt Learning with SAM for Few-shot Scanning Probe Microscope Image Segmentation

• Authors: Yao Shen, Ziwei Wei, Chunmeng Liu, Shuming Wei, Qi Zhao, Kaiyang Zeng, Guangyao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12562
• Pdf URL: https://arxiv.org/pdf/2410.12562
• Copy Paste: [[2410.12562]] Adaptive Prompt Learning with SAM for Few-shot Scanning Probe Microscope Image Segmentation(https://arxiv.org/abs/2410.12562)
• Keywords: segmentation
• Abstract: The Segment Anything Model (SAM) has demonstrated strong performance in image segmentation of natural scene images. However, its effectiveness diminishes markedly when applied to specific scientific domains, such as Scanning Probe Microscope (SPM) images. This decline in accuracy can be attributed to the distinct data distribution and limited availability of the data inherent in the scientific images. On the other hand, the acquisition of adequate SPM datasets is both time-intensive and laborious as well as skill-dependent. To address these challenges, we propose an Adaptive Prompt Learning with SAM (APL-SAM) framework tailored for few-shot SPM image segmentation. Our approach incorporates two key innovations to enhance SAM: 1) An Adaptive Prompt Learning module leverages few-shot embeddings derived from limited support set to learn adaptively central representatives, serving as visual prompts. This innovation eliminates the need for time-consuming online user interactions for providing prompts, such as exhaustively marking points and bounding boxes slice by slice; 2) A multi-source, multi-level mask decoder specifically designed for few-shot SPM image segmentation is introduced, which can effectively capture the correspondence between the support and query images. To facilitate comprehensive training and evaluation, we introduce a new dataset, SPM-Seg, curated for SPM image segmentation. Extensive experiments on this dataset reveal that the proposed APL-SAM framework significantly outperforms the original SAM, achieving over a 30% improvement in terms of Dice Similarity Coefficient with only one-shot guidance. Moreover, APL-SAM surpasses state-of-the-art few-shot segmentation methods and even fully supervised approaches in performance. Code and dataset used in this study will be made available upon acceptance.

Title: FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion

• Authors: Jiacheng Ruan, Yebin Yang, Zehao Lin, Feiyu Xiong, Zeyun Tang, Zhiyu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12564
• Pdf URL: https://arxiv.org/pdf/2410.12564
• Copy Paste: [[2410.12564]] FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion(https://arxiv.org/abs/2410.12564)
• Keywords: large language model
• Abstract: Benefiting from the revolutionary advances in large language models (LLMs) and foundational vision models, large vision-language models (LVLMs) have also made significant progress. However, current benchmarks focus on tasks that evaluating only a single aspect of LVLM capabilities (e.g., recognition, detection, understanding). These tasks fail to fully demonstrate LVLMs' potential in complex application scenarios. To comprehensively assess the performance of existing LVLMs, we propose a more challenging task called the Flow Text with Image Insertion task (FTII). This task requires LVLMs to simultaneously possess outstanding abilities in image comprehension, instruction understanding, and long-text interpretation. Specifically, given several text paragraphs and a set of candidate images, as the text paragraphs accumulate, the LVLMs are required to select the most suitable image from the candidates to insert after the corresponding paragraph. Constructing a benchmark for such a task is highly challenging, particularly in determining the sequence of flowing text and images. To address this challenge, we turn to professional news reports, which naturally contain a gold standard for image-text sequences. Based on this, we introduce the Flow Text with Image Insertion Benchmark (FTII-Bench), which includes 318 high-quality Chinese image-text news articles and 307 high-quality English image-text news articles, covering 10 different news domains. Using these 625 high-quality articles, we construct problems of two different types with multiple levels of difficulty. Furthermore, we establish two different evaluation pipelines based on the CLIP model and existing LVLMs. We evaluate 9 open-source and 2 closed-source LVLMs as well as 2 CLIP-based models. Results indicate that even the most advanced models (e.g., GPT-4o) face significant challenges when tackling the FTII task.

Title: Can We Reverse In-Context Knowledge Edits?

• Authors: Paul Youssef, Zhixue Zhao, Jörg Schlötterer, Christin Seifert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12586
• Pdf URL: https://arxiv.org/pdf/2410.12586
• Copy Paste: [[2410.12586]] Can We Reverse In-Context Knowledge Edits?(https://arxiv.org/abs/2410.12586)
• Keywords: large language model
• Abstract: In-context knowledge editing (IKE) enables efficient modification of large language model (LLM) outputs without parameter changes and at zero-cost. However, it can be misused to manipulate responses opaquely, e.g., insert misinformation or offensive content. Such malicious interventions could be incorporated into high-level wrapped APIs where the final input prompt is not shown to end-users. To address this issue, we investigate the detection and reversal of IKE-edits. First, we demonstrate that IKE-edits can be detected with high accuracy (F1 > 80\%) using only the top-10 output probabilities of the next token, even in a black-box setting, e.g. proprietary LLMs with limited output information. Further, we introduce the novel task of reversing IKE-edits using specially tuned reversal tokens. We explore using both continuous and discrete reversal tokens, achieving over 80\% accuracy in recovering original, unedited outputs across multiple LLMs. Our continuous reversal tokens prove particularly effective, with minimal impact on unedited prompts. Through analysis of output distributions, attention patterns, and token rankings, we provide insights into IKE's effects on LLMs and how reversal tokens mitigate them. This work represents a significant step towards enhancing LLM resilience against potential misuse of in-context editing, improving their transparency and trustworthiness.

Title: Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion

• Authors: Minkyoung Cho, Yulong Cao, Jiachen Sun, Qingzhao Zhang, Marco Pavone, Jeong Joon Park, Heng Yang, Z. Morley Mao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12592
• Pdf URL: https://arxiv.org/pdf/2410.12592
• Copy Paste: [[2410.12592]] Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion(https://arxiv.org/abs/2410.12592)
• Keywords: robust, fair
• Abstract: An important paradigm in 3D object detection is the use of multiple modalities to enhance accuracy in both normal and challenging conditions, particularly for long-tail scenarios. To address this, recent studies have explored two directions of adaptive approaches: MoE-based adaptive fusion, which struggles with uncertainties arising from distinct object configurations, and late fusion for output-level adaptive fusion, which relies on separate detection pipelines and limits comprehensive understanding. In this work, we introduce Cocoon, an object- and feature-level uncertainty-aware fusion framework. The key innovation lies in uncertainty quantification for heterogeneous representations, enabling fair comparison across modalities through the introduction of a feature aligner and a learnable surrogate ground truth, termed feature impression. We also define a training objective to ensure that their relationship provides a valid metric for uncertainty quantification. Cocoon consistently outperforms existing static and adaptive methods in both normal and challenging conditions, including those with natural and artificial corruptions. Furthermore, we show the validity and efficacy of our uncertainty metric across diverse datasets.

Title: On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs

• Authors: Herun Wan, Minnan Luo, Zhixiong Su, Guang Dai, Xiang Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12600
• Pdf URL: https://arxiv.org/pdf/2410.12600
• Copy Paste: [[2410.12600]] On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs(https://arxiv.org/abs/2410.12600)
• Keywords: defense, large language model
• Abstract: Evidence-enhanced detectors present remarkable abilities in identifying malicious social text with related evidence. However, the rise of large language models (LLMs) brings potential risks of evidence pollution to confuse detectors. This paper explores how to manipulate evidence, simulating potential misuse scenarios including basic pollution, and rephrasing or generating evidence by LLMs. To mitigate its negative impact, we propose three defense strategies from both the data and model sides, including machine-generated text detection, a mixture of experts, and parameter updating. Extensive experiments on four malicious social text detection tasks with ten datasets present that evidence pollution, especially the generate strategy, significantly compromises existing detectors. On the other hand, the defense strategies could mitigate evidence pollution, but they faced limitations for practical employment, such as the need for annotated data and huge inference costs. Further analysis illustrates that polluted evidence is of high quality, would compromise the model calibration, and could ensemble to amplify the negative impact.

Title: CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarization

• Authors: Yixi Ding, Jiaying Wu, Tongyao Zhu, Yanxia Qin, Qian Liu, Min-Yen Kan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12601
• Pdf URL: https://arxiv.org/pdf/2410.12601
• Copy Paste: [[2410.12601]] CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarization(https://arxiv.org/abs/2410.12601)
• Keywords: large language model
• Abstract: To broaden the dissemination of scientific knowledge to diverse audiences, scientific document summarization must simultaneously control multiple attributes such as length and empirical focus. However, existing research typically focuses on controlling single attributes, leaving the compositional control of multiple attributes underexplored. To address this gap, we introduce CCSBench, a benchmark for compositional controllable summarization in the scientific domain. Our benchmark enables fine-grained control over both explicit attributes (e.g., length), which are objective and straightforward, and implicit attributes (e.g., empirical focus), which are more subjective and conceptual. We conduct extensive experiments on GPT-4, LLaMA2, and other popular LLMs under various settings. Our findings reveal significant limitations in large language models' ability to balance trade-offs between control attributes, especially implicit ones that require deeper understanding and abstract reasoning.

Title: Advancing Web Browser Forensics: Critical Evaluation of Emerging Tools and Techniques

• Authors: Rishal Ravikesh Chand, Neeraj Anand Sharma, Muhammad Ashad Kabir
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.12605
• Pdf URL: https://arxiv.org/pdf/2410.12605
• Copy Paste: [[2410.12605]] Advancing Web Browser Forensics: Critical Evaluation of Emerging Tools and Techniques(https://arxiv.org/abs/2410.12605)
• Keywords: privacy, protect
• Abstract: As the use of web browsers continues to grow, the potential for cybercrime and web-related criminal activities also increases. Digital forensic investigators must understand how different browsers function and the critical areas to consider during web forensic analysis. Web forensics, a subfield of digital forensics, involves collecting and analyzing browser artifacts, such as browser history, search keywords, and downloads, which serve as potential evidence. While existing research has provided valuable insights, many studies focus on individual browsing modes or limited forensic scenarios, leaving gaps in understanding the full scope of data retention and recovery across different modes and browsers. This paper addresses these gaps by defining four browsing scenarios and critically analyzing browser artifacts across normal, private, and portable modes using various forensic tools. We define four browsing scenarios to perform a comprehensive evaluation of popular browsers -- Google Chrome, Mozilla Firefox, Brave, Tor, and Microsoft Edge -- by monitoring changes in key data storage areas such as cache files, cookies, browsing history, and local storage across different browsing modes. Overall, this paper contributes to a deeper understanding of browser forensic analysis and identifies key areas for enhancing privacy protection and forensic methodologies.

Title: Low-Rank Adversarial PGD Attack

• Authors: Dayana Savostianova, Emanuele Zangrando, Francesco Tudisco
• Subjects: cs.LG, cs.AI, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12607
• Pdf URL: https://arxiv.org/pdf/2410.12607
• Copy Paste: [[2410.12607]] Low-Rank Adversarial PGD Attack(https://arxiv.org/abs/2410.12607)
• Keywords: attack, robust
• Abstract: Adversarial attacks on deep neural network models have seen rapid development and are extensively used to study the stability of these networks. Among various adversarial strategies, Projected Gradient Descent (PGD) is a widely adopted method in computer vision due to its effectiveness and quick implementation, making it suitable for adversarial training. In this work, we observe that in many cases, the perturbations computed using PGD predominantly affect only a portion of the singular value spectrum of the original image, suggesting that these perturbations are approximately low-rank. Motivated by this observation, we propose a variation of PGD that efficiently computes a low-rank attack. We extensively validate our method on a range of standard models as well as robust models that have undergone adversarial training. Our analysis indicates that the proposed low-rank PGD can be effectively used in adversarial training due to its straightforward and fast implementation coupled with competitive performance. Notably, we find that low-rank PGD often performs comparably to, and sometimes even outperforms, the traditional full-rank PGD attack, while using significantly less memory.

Title: Not All Votes Count! Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning

• Authors: Vernon Y.H. Toh, Deepanway Ghosal, Soujanya Poria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12608
• Pdf URL: https://arxiv.org/pdf/2410.12608
• Copy Paste: [[2410.12608]] Not All Votes Count! Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning(https://arxiv.org/abs/2410.12608)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown increasing proficiency in solving mathematical reasoning problems. However, many current open-source LLMs often still make calculation and semantic understanding errors in their intermediate reasoning steps. In this work, we propose PROVE, a simple yet effective framework that uses program-based verification as a heuristic to filter out potentially incorrect reasoning paths before aggregating the final answers. Instead of relying on vanilla majority voting, our approach rejects solutions whose corresponding program outputs are inconsistent with the generated solution, aggregating only those validated by Python programs. We conducted extensive experiments on 13 open-source LLMs from various model families and sizes, ranging from 0.5B to 13B parameters, across seven math benchmarks. We demonstrate that PROVE consistently outperforms vanilla majority voting as a heuristic for solving mathematical reasoning tasks across all datasets and model sizes. Notably, PROVE increases accuracy on the GSM8K benchmark from 48.85% to 53.83% for Qwen2-0.5B-Instruct, from 65.66% to 73.01% for Llama-3.2-1B-Instruct, from 73.39% to 79.61% for Gemma-2-2b-it, and from 41.32% to 59.51% for Llama-2-7B-chat. Our codes are available at this https URL.

Title: Exploring Model Kinship for Merging Large Language Models

• Authors: Yedi Hu, Yunzhi Yao, Ningyu Zhang, Shumin Deng, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2410.12613
• Pdf URL: https://arxiv.org/pdf/2410.12613
• Copy Paste: [[2410.12613]] Exploring Model Kinship for Merging Large Language Models(https://arxiv.org/abs/2410.12613)
• Keywords: large language model
• Abstract: Model merging has become one of the key technologies for enhancing the capabilities and efficiency of Large Language Models (LLMs). However, our understanding of the expected performance gains and principles when merging any two models remains limited. In this work, we introduce model kinship, the degree of similarity or relatedness between LLMs, analogous to biological evolution. With comprehensive empirical analysis, we find that there is a certain relationship between model kinship and the performance gains after model merging, which can help guide our selection of candidate models. Inspired by this, we propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can yield better performance on benchmark datasets. Specifically, we discover that using model kinship as a criterion can assist us in continuously performing model merging, alleviating the degradation (local optima) in model evolution, whereas model kinship can serve as a guide to escape these traps. Code is available at this https URL.

Title: Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning

• Authors: Ruimeng Ye, Yang Xiao, Bo Hui
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12621
• Pdf URL: https://arxiv.org/pdf/2410.12621
• Copy Paste: [[2410.12621]] Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning(https://arxiv.org/abs/2410.12621)
• Keywords: large language model
• Abstract: As large language models (LLMs) continue to advance, ensuring their alignment with human values becomes increasingly critical. Traditional alignment methods heavily rely on human feedback to fine-tune models. With the emergence of superhuman models whose outputs may surpass human understanding, evaluating and aligning these models using human judgments poses significant challenges. To address the challenges, recent works use weak supervisors to elicit knowledge from much stronger models. However, there are important disanalogies between the empirical setup in the existing works and the genuine goal of alignment. We remark that existing works investigate the phenomenon of weak-to-strong generation in analogous setup (i.e., binary classification), rather than practical alignment-relevant tasks (e.g., safety). In this paper, we bridge this gap by extending weak-to-strong generation to the context of practical alignment. We empirically demonstrate the widespread phenomenon of weak-to-strong generation in three complicated alignment tasks: safety, toxicity, and legal reasoning}. Furthermore, we explore efficient strategies for improving alignment performance to enhance the quality of model outcomes. Lastly, we summarize and analyze the challenges and potential solutions in regard to specific alignment tasks, which we hope to catalyze the research progress on the topic of weak-to-strong generalization. Our code is released at this https URL.

Title: DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception

• Authors: Zhiyuan Zhao, Hengrui Kang, Bin Wang, Conghui He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12628
• Pdf URL: https://arxiv.org/pdf/2410.12628
• Copy Paste: [[2410.12628]] DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception(https://arxiv.org/abs/2410.12628)
• Keywords: robust
• Abstract: Document Layout Analysis is crucial for real-world document understanding systems, but it encounters a challenging trade-off between speed and accuracy: multimodal methods leveraging both text and visual features achieve higher accuracy but suffer from significant latency, whereas unimodal methods relying solely on visual features offer faster processing speeds at the expense of accuracy. To address this dilemma, we introduce DocLayout-YOLO, a novel approach that enhances accuracy while maintaining speed advantages through document-specific optimizations in both pre-training and model design. For robust document pre-training, we introduce the Mesh-candidate BestFit algorithm, which frames document synthesis as a two-dimensional bin packing problem, generating the large-scale, diverse DocSynth-300K dataset. Pre-training on the resulting DocSynth-300K dataset significantly improves fine-tuning performance across various document types. In terms of model optimization, we propose a Global-to-Local Controllable Receptive Module that is capable of better handling multi-scale variations of document elements. Furthermore, to validate performance across different document types, we introduce a complex and challenging benchmark named DocStructBench. Extensive experiments on downstream datasets demonstrate that DocLayout-YOLO excels in both speed and accuracy. Code, data, and models are available at this https URL.

Title: Constrained Posterior Sampling: Time Series Generation with Hard Constraints

• Authors: Sai Shankar Narasimhan, Shubhankar Agarwal, Litu Rout, Sanjay Shakkottai, Sandeep P. Chinchali
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.12652
• Pdf URL: https://arxiv.org/pdf/2410.12652
• Copy Paste: [[2410.12652]] Constrained Posterior Sampling: Time Series Generation with Hard Constraints(https://arxiv.org/abs/2410.12652)
• Keywords: privacy, protect, diffusion
• Abstract: Generating realistic time series samples is crucial for stress-testing models and protecting user privacy by using synthetic data. In engineering and safety-critical applications, these samples must meet certain hard constraints that are domain-specific or naturally imposed by physics or nature. Consider, for example, generating electricity demand patterns with constraints on peak demand times. This can be used to stress-test the functioning of power grids during adverse weather conditions. Existing approaches for generating constrained time series are either not scalable or degrade sample quality. To address these challenges, we introduce Constrained Posterior Sampling (CPS), a diffusion-based sampling algorithm that aims to project the posterior mean estimate into the constraint set after each denoising update. Notably, CPS scales to a large number of constraints (~100) without requiring additional training. We provide theoretical justifications highlighting the impact of our projection step on sampling. Empirically, CPS outperforms state-of-the-art methods in sample quality and similarity to real time series by around 10% and 42%, respectively, on real-world stocks, traffic, and air quality datasets.

Title: Evaluating Morphological Compositional Generalization in Large Language Models

• Authors: Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Lonneke van der Plas, Duygu Ataman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12656
• Pdf URL: https://arxiv.org/pdf/2410.12656
• Copy Paste: [[2410.12656]] Evaluating Morphological Compositional Generalization in Large Language Models(https://arxiv.org/abs/2410.12656)
• Keywords: generative, large language model
• Abstract: Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabilities remain questionable, raising doubts about whether these models learn language similarly to humans. While humans exhibit compositional generalization and linguistic creativity in language use, the extent to which LLMs replicate these abilities, particularly in morphology, is under-explored. In this work, we systematically investigate the morphological generalization abilities of LLMs through the lens of compositionality. We define morphemes as compositional primitives and design a novel suite of generative and discriminative tasks to assess morphological productivity and systematicity. Focusing on agglutinative languages such as Turkish and Finnish, we evaluate several state-of-the-art instruction-finetuned multilingual models, including GPT-4 and Gemini. Our analysis shows that LLMs struggle with morphological compositional generalization particularly when applied to novel word roots, with performance declining sharply as morphological complexity increases. While models can identify individual morphological combinations better than chance, their performance lacks systematicity, leading to significant accuracy gaps compared to humans.

Title: Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models

• Authors: Shicheng Xu, Liang Pang, Yunchang Zhu, Huawei Shen, Xueqi Cheng
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12662
• Pdf URL: https://arxiv.org/pdf/2410.12662
• Copy Paste: [[2410.12662]] Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models(https://arxiv.org/abs/2410.12662)
• Keywords: transformer
• Abstract: Vision-language alignment in Large Vision-Language Models (LVLMs) successfully enables LLMs to understand visual input. However, we find that existing vision-language alignment methods fail to transfer the existing safety mechanism for text in LLMs to vision, which leads to vulnerabilities in toxic image. To explore the cause of this problem, we give the insightful explanation of where and how the safety mechanism of LVLMs operates and conduct comparative analysis between text and vision. We find that the hidden states at the specific transformer layers play a crucial role in the successful activation of safety mechanism, while the vision-language alignment at hidden states level in current methods is insufficient. This results in a semantic shift for input images compared to text in hidden states, therefore misleads the safety mechanism. To address this, we propose a novel Text-Guided vision-language Alignment method (TGA) for LVLMs. TGA retrieves the texts related to input vision and uses them to guide the projection of vision into the hidden states space in LLMs. Experiments show that TGA not only successfully transfers the safety mechanism for text in basic LLMs to vision in vision-language alignment for LVLMs without any safety fine-tuning on the visual modality but also maintains the general performance on various vision tasks (Safe and Good).

Title: 3DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generation

• Authors: Dewei Zhou, Ji Xie, Zongxin Yang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12669
• Pdf URL: https://arxiv.org/pdf/2410.12669
• Copy Paste: [[2410.12669]] 3DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generation(https://arxiv.org/abs/2410.12669)
• Keywords: robust
• Abstract: The increasing demand for controllable outputs in text-to-image generation has spurred advancements in multi-instance generation (MIG), allowing users to define both instance layouts and attributes. However, unlike image-conditional generation methods such as ControlNet, MIG techniques have not been widely adopted in state-of-the-art models like SD2 and SDXL, primarily due to the challenge of building robust renderers that simultaneously handle instance positioning and attribute rendering. In this paper, we introduce Depth-Driven Decoupled Instance Synthesis (3DIS), a novel framework that decouples the MIG process into two stages: (i) generating a coarse scene depth map for accurate instance positioning and scene composition, and (ii) rendering fine-grained attributes using pre-trained ControlNet on any foundational model, without additional training. Our 3DIS framework integrates a custom adapter into LDM3D for precise depth-based layouts and employs a finetuning-free method for enhanced instance-level attribute rendering. Extensive experiments on COCO-Position and COCO-MIG benchmarks demonstrate that 3DIS significantly outperforms existing methods in both layout precision and attribute rendering. Notably, 3DIS offers seamless compatibility with diverse foundational models, providing a robust, adaptable solution for advanced multi-instance generation. The code is available at: this https URL.

Title: New Paradigm of Adversarial Training: Breaking Inherent Trade-Off between Accuracy and Robustness via Dummy Classes

• Authors: Yanyun Wang, Li Liu, Zi Liang, Qingqing Ye, Haibo Hu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12671
• Pdf URL: https://arxiv.org/pdf/2410.12671
• Copy Paste: [[2410.12671]] New Paradigm of Adversarial Training: Breaking Inherent Trade-Off between Accuracy and Robustness via Dummy Classes(https://arxiv.org/abs/2410.12671)
• Keywords: robust
• Abstract: Adversarial Training (AT) is one of the most effective methods to enhance the robustness of DNNs. However, existing AT methods suffer from an inherent trade-off between adversarial robustness and clean accuracy, which seriously hinders their real-world deployment. While this problem has been widely studied within the current AT paradigm, existing AT methods still typically experience a reduction in clean accuracy by over 10% to date, without significant improvements in robustness compared with simple baselines like PGD-AT. This inherent trade-off raises a question: whether the current AT paradigm, which assumes to learn the corresponding benign and adversarial samples as the same class, inappropriately combines clean and robust objectives that may be essentially inconsistent. In this work, we surprisingly reveal that up to 40% of CIFAR-10 adversarial samples always fail to satisfy such an assumption across various AT methods and robust models, explicitly indicating the improvement room for the current AT paradigm. Accordingly, to relax the tension between clean and robust learning derived from this overstrict assumption, we propose a new AT paradigm by introducing an additional dummy class for each original class, aiming to accommodate the hard adversarial samples with shifted distribution after perturbation. The robustness w.r.t. these adversarial samples can be achieved by runtime recovery from the predicted dummy classes to their corresponding original ones, eliminating the compromise with clean learning. Building on this new paradigm, we propose a novel plug-and-play AT technology named DUmmy Classes-based Adversarial Training (DUCAT). Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that the DUCAT concurrently improves clean accuracy and adversarial robustness compared with state-of-the-art benchmarks, effectively breaking the existing inherent trade-off.

Title: Optimizing Multi-Task Learning for Accurate Spacecraft Pose Estimation

• Authors: Francesco Evangelisti, Francesco Rossi, Tobia Giani, Ilaria Bloise, Mattia Varile
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12679
• Pdf URL: https://arxiv.org/pdf/2410.12679
• Copy Paste: [[2410.12679]] Optimizing Multi-Task Learning for Accurate Spacecraft Pose Estimation(https://arxiv.org/abs/2410.12679)
• Keywords: robust, segmentation
• Abstract: Accurate satellite pose estimation is crucial for autonomous guidance, navigation, and control (GNC) systems in in-orbit servicing (IOS) missions. This paper explores the impact of different tasks within a multi-task learning (MTL) framework for satellite pose estimation using monocular images. By integrating tasks such as direct pose estimation, keypoint prediction, object localization, and segmentation into a single network, the study aims to evaluate the reciprocal influence between tasks by testing different multi-task configurations thanks to the modularity of the convolutional neural network (CNN) used in this work. The trends of mutual bias between the analyzed tasks are found by employing different weighting strategies to further test the robustness of the findings. A synthetic dataset was developed to train and test the MTL network. Results indicate that direct pose estimation and heatmap-based pose estimation positively influence each other in general, while both the bounding box and segmentation tasks do not provide significant contributions and tend to degrade the overall estimation accuracy.

Title: Automatic Mapping of Anatomical Landmarks from Free-Text Using Large Language Models: Insights from Llama-2

• Authors: Mohamad Abdi, Gerardo Hemosillo Valadez, Halid Ziya Yerebakan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12686
• Pdf URL: https://arxiv.org/pdf/2410.12686
• Copy Paste: [[2410.12686]] Automatic Mapping of Anatomical Landmarks from Free-Text Using Large Language Models: Insights from Llama-2(https://arxiv.org/abs/2410.12686)
• Keywords: robust, generative, large language model
• Abstract: Anatomical landmarks are vital in medical imaging for navigation and anomaly detection. Modern large language models (LLMs), like Llama-2, offer promise for automating the mapping of these landmarks in free-text radiology reports to corresponding positions in image data. Recent studies propose LLMs may develop coherent representations of generative processes. Motivated by these insights, we investigated whether LLMs accurately represent the spatial positions of anatomical landmarks. Through experiments with Llama-2 models, we found that they can linearly represent anatomical landmarks in space with considerable robustness to different prompts. These results underscore the potential of LLMs to enhance the efficiency and accuracy of medical imaging workflows.

Title: VividMed: Vision Language Model with Versatile Visual Grounding for Medicine

• Authors: Lingxiao Luo, Bingda Tang, Xuanzhong Chen, Rong Han, Ting Chen
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12694
• Pdf URL: https://arxiv.org/pdf/2410.12694
• Copy Paste: [[2410.12694]] VividMed: Vision Language Model with Versatile Visual Grounding for Medicine(https://arxiv.org/abs/2410.12694)
• Keywords: segmentation
• Abstract: Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable promise in generating visually grounded responses. However, their application in the medical domain is hindered by unique challenges. For instance, most VLMs rely on a single method of visual grounding, whereas complex medical tasks demand more versatile approaches. Additionally, while most VLMs process only 2D images, a large portion of medical images are 3D. The lack of medical data further compounds these obstacles. To address these challenges, we present VividMed, a vision language model with versatile visual grounding for medicine. Our model supports generating both semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. We design a three-stage training procedure and an automatic data synthesis pipeline based on open datasets and models. Besides visual grounding tasks, VividMed also excels in other common downstream tasks, including Visual Question Answering (VQA) and report generation. Ablation studies empirically show that the integration of visual grounding ability leads to improved performance on these tasks. Our code is publicly available at this https URL.

Title: MultiCamCows2024 -- A Multi-view Image Dataset for AI-driven Holstein-Friesian Cattle Re-Identification on a Working Farm

• Authors: Phoenix Yu, Tilo Burghardt, Andrew W Dowsey, Neill W Campbell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12695
• Pdf URL: https://arxiv.org/pdf/2410.12695
• Copy Paste: [[2410.12695]] MultiCamCows2024 -- A Multi-view Image Dataset for AI-driven Holstein-Friesian Cattle Re-Identification on a Working Farm(https://arxiv.org/abs/2410.12695)
• Keywords: biometric
• Abstract: We present MultiCamCows2024, a farm-scale image dataset filmed across multiple cameras for the biometric identification of individual Holstein-Friesian cattle exploiting their unique black and white coat-patterns. Captured by three ceiling-mounted visual sensors covering adjacent barn areas over seven days on a working dairy farm, the dataset comprises 101, 329 images of 90 cows, plus the underlying original CCTV footage. The dataset is provided alongside full computer vision recognition baselines, that is both a supervised and self-supervised learning framework for individual cow identification trained on cattle tracklets. We report a performance above 96% single image identification accuracy from the dataset and demonstrate that combining data from multiple cameras during learning enhances self-supervised identification. We show that our framework enables fully automatic cattle identification, barring only the simple human verification of tracklet integrity during data collection. Crucially, our study highlights that multi-camera, supervised and self-supervised components in tandem not only deliver highly accurate individual cow identification but also achieve this efficiently with no labelling of cattle identities by humans at all. We argue that this improvement in efficacy has practical implications for livestock management, behaviour analysis, and agricultural monitoring. For full reproducibility and practical ease of use, we publish all key software and code including re-identification components and the species detector with this paper.

Title: AdaptiveDrag: Semantic-Driven Dragging on Diffusion-Based Image Editing

• Authors: DuoSheng Chen, Binghui Chen, Yifeng Geng, Liefeng Bo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12696
• Pdf URL: https://arxiv.org/pdf/2410.12696
• Copy Paste: [[2410.12696]] AdaptiveDrag: Semantic-Driven Dragging on Diffusion-Based Image Editing(https://arxiv.org/abs/2410.12696)
• Keywords: diffusion, generative, segmentation
• Abstract: Recently, several point-based image editing methods (e.g., DragDiffusion, FreeDrag, DragNoise) have emerged, yielding precise and high-quality results based on user instructions. However, these methods often make insufficient use of semantic information, leading to less desirable results. In this paper, we proposed a novel mask-free point-based image editing method, AdaptiveDrag, which provides a more flexible editing approach and generates images that better align with user intent. Specifically, we design an auto mask generation module using super-pixel division for user-friendliness. Next, we leverage a pre-trained diffusion model to optimize the latent, enabling the dragging of features from handle points to target points. To ensure a comprehensive connection between the input image and the drag process, we have developed a semantic-driven optimization. We design adaptive steps that are supervised by the positions of the points and the semantic regions derived from super-pixel segmentation. This refined optimization process also leads to more realistic and accurate drag results. Furthermore, to address the limitations in the generative consistency of the diffusion model, we introduce an innovative corresponding loss during the sampling process. Building on these effective designs, our method delivers superior generation results using only the single input image and the handle-target point pairs. Extensive experiments have been conducted and demonstrate that the proposed method outperforms others in handling various drag instructions (e.g., resize, movement, extension) across different domains (e.g., animals, human face, land space, clothing).

Title: Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimization

• Authors: Xingqi Wang, Xiaoyuan Yi, Xing Xie, Jia Jia
• Subjects: cs.CV, cs.AI, cs.CY, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.12700
• Pdf URL: https://arxiv.org/pdf/2410.12700
• Copy Paste: [[2410.12700]] Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimization(https://arxiv.org/abs/2410.12700)
• Keywords: diffusion, large language model
• Abstract: Recent advancements in diffusion models trained on large-scale data have enabled the generation of indistinguishable human-level images, yet they often produce harmful content misaligned with human values, e.g., social bias, and offensive content. Despite extensive research on Large Language Models (LLMs), the challenge of Text-to-Image (T2I) model alignment remains largely unexplored. Addressing this problem, we propose LiVO (Lightweight Value Optimization), a novel lightweight method for aligning T2I models with human values. LiVO only optimizes a plug-and-play value encoder to integrate a specified value principle with the input prompt, allowing the control of generated images over both semantics and values. Specifically, we design a diffusion model-tailored preference optimization loss, which theoretically approximates the Bradley-Terry model used in LLM alignment but provides a more flexible trade-off between image quality and value conformity. To optimize the value encoder, we also develop a framework to automatically construct a text-image preference dataset of 86k (prompt, aligned image, violating image, value principle) samples. Without updating most model parameters and through adaptive value selection from the input prompt, LiVO significantly reduces harmful outputs and achieves faster convergence, surpassing several strong baselines and taking an initial step towards ethically aligned T2I models.

Title: Neural-based Control for CubeSat Docking Maneuvers

• Authors: Matteo Stoisa, Federica Paganelli Azza, Luca Romanelli, Mattia Varile
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12703
• Pdf URL: https://arxiv.org/pdf/2410.12703
• Copy Paste: [[2410.12703]] Neural-based Control for CubeSat Docking Maneuvers(https://arxiv.org/abs/2410.12703)
• Keywords: robust
• Abstract: Autonomous Rendezvous and Docking (RVD) have been extensively studied in recent years, addressing the stringent requirements of spacecraft dynamics variations and the limitations of GNC systems. This paper presents an innovative approach employing Artificial Neural Networks (ANN) trained through Reinforcement Learning (RL) for autonomous spacecraft guidance and control during the final phase of the rendezvous maneuver. The proposed strategy is easily implementable onboard and offers fast adaptability and robustness to disturbances by learning control policies from experience rather than relying on predefined models. Extensive Monte Carlo simulations within a relevant environment are conducted in 6DoF settings to validate our approach, along with hardware tests that demonstrate deployment feasibility. Our findings highlight the efficacy of RL in assuring the adaptability and efficiency of spacecraft RVD, offering insights into future mission expectations.

Title: Sarcasm Detection in a Less-Resourced Language

• Authors: Lazar Đoković, Marko Robnik-Šikonja
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12704
• Pdf URL: https://arxiv.org/pdf/2410.12704
• Copy Paste: [[2410.12704]] Sarcasm Detection in a Less-Resourced Language(https://arxiv.org/abs/2410.12704)
• Keywords: transformer, generative
• Abstract: The sarcasm detection task in natural language processing tries to classify whether an utterance is sarcastic or not. It is related to sentiment analysis since it often inverts surface sentiment. Because sarcastic sentences are highly dependent on context, and they are often accompanied by various non-verbal cues, the task is challenging. Most of related work focuses on high-resourced languages like English. To build a sarcasm detection dataset for a less-resourced language, such as Slovenian, we leverage two modern techniques: a machine translation specific medium-size transformer model, and a very large generative language model. We explore the viability of translated datasets and how the size of a pretrained transformer affects its ability to detect sarcasm. We train ensembles of detection models and evaluate models' performance. The results show that larger models generally outperform smaller ones and that ensembling can slightly improve sarcasm detection performance. Our best ensemble approach achieves an $\text{F}_1$-score of 0.765 which is close to annotators' agreement in the source language.

Title: WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation

• Authors: João Matos, Shan Chen, Siena Placino, Yingya Li, Juan Carlos Climent Pardo, Daphna Idan, Takeshi Tohyama, David Restrepo, Luis F. Nakayama, Jose M. M. Pascual-Leone, Guergana Savova, Hugo Aerts, Leo A. Celi, A. Ian Wong, Danielle S. Bitterman, Jack Gallifant
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12722
• Pdf URL: https://arxiv.org/pdf/2410.12722
• Copy Paste: [[2410.12722]] WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation(https://arxiv.org/abs/2410.12722)
• Keywords: robust, fair
• Abstract: Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fairness. Multiple-choice question and answer (QA) datasets derived from national medical examinations have long served as valuable evaluation tools, but existing datasets are largely text-only and available in a limited subset of languages and countries. To address these challenges, we present WorldMedQA-V, an updated multilingual, multimodal benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V includes 568 labeled multiple-choice QAs paired with 568 medical images from four countries (Brazil, Israel, Japan, and Spain), covering original languages and validated English translations by native clinicians, respectively. Baseline performance for common open- and closed-source models are provided in the local language and English translations, and with and without images provided to the model. The WorldMedQA-V benchmark aims to better match AI systems to the diverse healthcare environments in which they are deployed, fostering more equitable, effective, and representative applications.

Title: Transformer based super-resolution downscaling for regional reanalysis: Full domain vs tiling approaches

• Authors: Antonio Pérez, Mario Santa Cruz, Daniel San Martín, José Manuel Gutiérrez
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12728
• Pdf URL: https://arxiv.org/pdf/2410.12728
• Copy Paste: [[2410.12728]] Transformer based super-resolution downscaling for regional reanalysis: Full domain vs tiling approaches(https://arxiv.org/abs/2410.12728)
• Keywords: transformer
• Abstract: Super-resolution (SR) is a promising cost-effective downscaling methodology for producing high-resolution climate information from coarser counterparts. A particular application is downscaling regional reanalysis outputs (predictand) from the driving global counterparts (predictor). This study conducts an intercomparison of various SR downscaling methods focusing on temperature and using the CERRA reanalysis (5.5 km resolution, produced with a regional atmospheric model driven by ERA5) as example. The method proposed in this work is the Swin transformer and two alternative methods are used as benchmark (fully convolutional U-Net and convolutional and dense DeepESD) as well as the simple bicubic interpolation. We compare two approaches, the standard one using the full domain as input and a more scalable tiling approach, dividing the full domain into tiles that are used as input. The methods are trained to downscale CERRA surface temperature, based on temperature information from the driving ERA5; in addition, the tiling approach includes static orographic information. We show that the tiling approach, which requires spatial transferability, comes at the cost of a lower performance (although it outperforms some full-domain benchmarks), but provides an efficient scalable solution that allows SR reduction on a pan-European scale and is valuable for real-time applications.

Title: Counterfactual Generative Modeling with Variational Causal Inference

• Authors: Yulun Wu, Louie McConnell, Claudia Iriondo
• Subjects: cs.LG, cs.AI, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12730
• Pdf URL: https://arxiv.org/pdf/2410.12730
• Copy Paste: [[2410.12730]] Counterfactual Generative Modeling with Variational Causal Inference(https://arxiv.org/abs/2410.12730)
• Keywords: generative
• Abstract: Estimating an individual's potential outcomes under counterfactual treatments is a challenging task for traditional causal inference and supervised learning approaches when the outcome is high-dimensional (e.g. gene expressions, facial images) and covariates are relatively limited. In this case, to predict one's outcomes under counterfactual treatments, it is crucial to leverage individual information contained in its high-dimensional observed outcome in addition to the covariates. Prior works using variational inference in counterfactual generative modeling have been focusing on neural adaptations and model variants within the conditional variational autoencoder formulation, which we argue is fundamentally ill-suited to the notion of counterfactual in causal inference. In this work, we present a novel variational Bayesian causal inference framework and its theoretical backings to properly handle counterfactual generative modeling tasks, through which we are able to conduct counterfactual supervision end-to-end during training without any counterfactual samples, and encourage latent disentanglement that aids the correct identification of causal effect in counterfactual generations. In experiments, we demonstrate the advantage of our framework compared to state-of-the-art models in counterfactual generative modeling on multiple benchmarks.

Title: CREAM: Consistency Regularized Self-Rewarding Language Models

• Authors: Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, Huaxiu Yao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12735
• Pdf URL: https://arxiv.org/pdf/2410.12735
• Copy Paste: [[2410.12735]] CREAM: Consistency Regularized Self-Rewarding Language Models(https://arxiv.org/abs/2410.12735)
• Keywords: large language model
• Abstract: Recent self-rewarding large language models (LLM) have successfully applied LLM-as-a-Judge to iteratively improve the alignment performance without the need of human annotations for preference data. These methods commonly utilize the same LLM to act as both the policy model (which generates responses) and the reward model (which scores and ranks those responses). The ranked responses are then used as preference pairs to train the LLM via direct alignment technologies (e.g. DPO). However, it is noteworthy that throughout this process, there is no guarantee of accuracy in the rewarding and ranking, which is critical for ensuring accurate rewards and high-quality preference data. Empirical results from relatively small LLMs (e.g., 7B parameters) also indicate that improvements from self-rewarding may diminish after several iterations in certain situations, which we hypothesize is due to accumulated bias in the reward system. This bias can lead to unreliable preference data for training the LLM. To address this issue, we first formulate and analyze the generalized iterative preference fine-tuning framework for self-rewarding language model. We then introduce the regularization to this generalized framework to mitigate the overconfident preference labeling in the self-rewarding process. Based on this theoretical insight, we propose a Consistency Regularized sElf-rewarding lAnguage Model (CREAM) that leverages the rewarding consistency across different iterations to regularize the self-rewarding training, helping the model to learn from more reliable preference data. With this explicit regularization, our empirical results demonstrate the superiority of CREAM in improving both reward consistency and alignment performance. The code is publicly available at this https URL.

Title: Comparative Analysis of Extrinsic Factors for NER in French

• Authors: Grace Yang, Zhiyi Li, Yandong Liu, Jungyeul Park
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12750
• Pdf URL: https://arxiv.org/pdf/2410.12750
• Copy Paste: [[2410.12750]] Comparative Analysis of Extrinsic Factors for NER in French(https://arxiv.org/abs/2410.12750)
• Keywords: extraction
• Abstract: Named entity recognition (NER) is a crucial task that aims to identify structured information, which is often replete with complex, technical terms and a high degree of variability. Accurate and reliable NER can facilitate the extraction and analysis of important information. However, NER for other than English is challenging due to limited data availability, as the high expertise, time, and expenses are required to annotate its data. In this paper, by using the limited data, we explore various factors including model structure, corpus annotation scheme and data augmentation techniques to improve the performance of a NER model for French. Our experiments demonstrate that these approaches can significantly improve the model's F1 score from original CRF score of 62.41 to 79.39. Our findings suggest that considering different extrinsic factors and combining these techniques is a promising approach for improving NER performance where the size of data is limited.

Title: StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples

• Authors: Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, Chris Callison-Burch
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12757
• Pdf URL: https://arxiv.org/pdf/2410.12757
• Copy Paste: [[2410.12757]] StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples(https://arxiv.org/abs/2410.12757)
• Keywords: large language model
• Abstract: Style representations aim to embed texts with similar writing styles closely and texts with different styles far apart, regardless of content. However, the contrastive triplets often used for training these representations may vary in both style and content, leading to potential content leakage in the representations. We introduce StyleDistance, a novel approach to training stronger content-independent style embeddings. We use a large language model to create a synthetic dataset of near-exact paraphrases with controlled style variations, and produce positive and negative examples across 40 distinct style features for precise contrastive learning. We assess the quality of our synthetic data and embeddings through human and automatic evaluations. StyleDistance enhances the content-independence of style embeddings, which generalize to real-world benchmarks and outperform leading style representations in downstream applications. Our model can be found at this https URL .

Title: Unitary Multi-Margin BERT for Robust Natural Language Processing

• Authors: Hao-Yuan Chang, Kang L. Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12759
• Pdf URL: https://arxiv.org/pdf/2410.12759
• Copy Paste: [[2410.12759]] Unitary Multi-Margin BERT for Robust Natural Language Processing(https://arxiv.org/abs/2410.12759)
• Keywords: protect, defense, attack, robust, transformer
• Abstract: Recent developments in adversarial attacks on deep learning leave many mission-critical natural language processing (NLP) systems at risk of exploitation. To address the lack of computationally efficient adversarial defense methods, this paper reports a novel, universal technique that drastically improves the robustness of Bidirectional Encoder Representations from Transformers (BERT) by combining the unitary weights with the multi-margin loss. We discover that the marriage of these two simple ideas amplifies the protection against malicious interference. Our model, the unitary multi-margin BERT (UniBERT), boosts post-attack classification accuracies significantly by 5.3% to 73.8% while maintaining competitive pre-attack accuracies. Furthermore, the pre-attack and post-attack accuracy tradeoff can be adjusted via a single scalar parameter to best fit the design requirements for the target applications.

Title: SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation

• Authors: Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, Mohit Bansal
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12761
• Pdf URL: https://arxiv.org/pdf/2410.12761
• Copy Paste: [[2410.12761]] SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation(https://arxiv.org/abs/2410.12761)
• Keywords: robust, diffusion
• Abstract: Recent advances in diffusion models have significantly enhanced their ability to generate high-quality images and videos, but they have also increased the risk of producing unsafe content. Existing unlearning/editing-based methods for safe generation remove harmful concepts from models but face several challenges: (1) They cannot instantly remove harmful concepts without training. (2) Their safe generation capabilities depend on collected training data. (3) They alter model weights, risking degradation in quality for content unrelated to toxic concepts. To address these, we propose SAFREE, a novel, training-free approach for safe T2I and T2V, that does not alter the model's weights. Specifically, we detect a subspace corresponding to a set of toxic concepts in the text embedding space and steer prompt embeddings away from this subspace, thereby filtering out harmful content while preserving intended semantics. To balance the trade-off between filtering toxicity and preserving safe concepts, SAFREE incorporates a novel self-validating filtering mechanism that dynamically adjusts the denoising steps when applying the filtered embeddings. Additionally, we incorporate adaptive re-attention mechanisms within the diffusion latent space to selectively diminish the influence of features related to toxic concepts at the pixel level. In the end, SAFREE ensures coherent safety checking, preserving the fidelity, quality, and safety of the output. SAFREE achieves SOTA performance in suppressing unsafe content in T2I generation compared to training-free baselines and effectively filters targeted concepts while maintaining high-quality images. It also shows competitive results against training-based methods. We extend SAFREE to various T2I backbones and T2V tasks, showcasing its flexibility and generalization. SAFREE provides a robust and adaptable safeguard for ensuring safe visual generation.

Title: Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Information

• Authors: Yingya Li, Timothy Miller, Steven Bethard, Guergana Savova
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.12774
• Pdf URL: https://arxiv.org/pdf/2410.12774
• Copy Paste: [[2410.12774]] Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Information(https://arxiv.org/abs/2410.12774)
• Keywords: large language model
• Abstract: The success of multi-task learning can depend heavily on which tasks are grouped together. Naively grouping all tasks or a random set of tasks can result in negative transfer, with the multi-task models performing worse than single-task models. Though many efforts have been made to identify task groupings and to measure the relatedness among different tasks, it remains a challenging research topic to define a metric to identify the best task grouping out of a pool of many potential task combinations. We propose a metric of task relatedness based on task difficulty measured by pointwise V-usable information (PVI). PVI is a recently proposed metric to estimate how much usable information a dataset contains given a model. We hypothesize that tasks with not statistically different PVI estimates are similar enough to benefit from the joint learning process. We conduct comprehensive experiments to evaluate the feasibility of this metric for task grouping on 15 NLP datasets in the general, biomedical, and clinical domains. We compare the results of the joint learners against single learners, existing baseline methods, and recent large language models, including Llama 2 and GPT-4. The results show that by grouping tasks with similar PVI estimates, the joint learners yielded competitive results with fewer total parameters, with consistent performance across domains.

Title: Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts

• Authors: Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, Min Lin
• Subjects: cs.CV, cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12777
• Pdf URL: https://arxiv.org/pdf/2410.12777
• Copy Paste: [[2410.12777]] Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts(https://arxiv.org/abs/2410.12777)
• Keywords: diffusion
• Abstract: With the rapid progress of diffusion-based content generation, significant efforts are being made to unlearn harmful or copyrighted concepts from pretrained diffusion models (DMs) to prevent potential model misuse. However, it is observed that even when DMs are properly unlearned before release, malicious finetuning can compromise this process, causing DMs to relearn the unlearned concepts. This occurs partly because certain benign concepts (e.g., "skin") retained in DMs are related to the unlearned ones (e.g., "nudity"), facilitating their relearning via finetuning. To address this, we propose meta-unlearning on DMs. Intuitively, a meta-unlearned DM should behave like an unlearned DM when used as is; moreover, if the meta-unlearned DM undergoes malicious finetuning on unlearned concepts, the related benign concepts retained within it will be triggered to self-destruct, hindering the relearning of unlearned concepts. Our meta-unlearning framework is compatible with most existing unlearning methods, requiring only the addition of an easy-to-implement meta objective. We validate our approach through empirical experiments on meta-unlearning concepts from Stable Diffusion models (SD-v1-4 and SDXL), supported by extensive ablation studies. Our code is available at this https URL.

Title: Geometry-Aware Generative Autoencoders for Warped Riemannian Metric Learning and Generative Modeling on Data Manifolds

• Authors: Xingzhi Sun, Danqi Liao, Kincaid MacDonald, Yanlei Zhang, Chen Liu, Guillaume Huguet, Guy Wolf, Ian Adelstein, Tim G. J. Rudner, Smita Krishnaswamy
• Subjects: cs.LG, math.DG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12779
• Pdf URL: https://arxiv.org/pdf/2410.12779
• Copy Paste: [[2410.12779]] Geometry-Aware Generative Autoencoders for Warped Riemannian Metric Learning and Generative Modeling on Data Manifolds(https://arxiv.org/abs/2410.12779)
• Keywords: generative
• Abstract: Rapid growth of high-dimensional datasets in fields such as single-cell RNA sequencing and spatial genomics has led to unprecedented opportunities for scientific discovery, but it also presents unique computational and statistical challenges. Traditional methods struggle with geometry-aware data generation, interpolation along meaningful trajectories, and transporting populations via feasible paths. To address these issues, we introduce Geometry-Aware Generative Autoencoder (GAGA), a novel framework that combines extensible manifold learning with generative modeling. GAGA constructs a neural network embedding space that respects the intrinsic geometries discovered by manifold learning and learns a novel warped Riemannian metric on the data space. This warped metric is derived from both the points on the data manifold and negative samples off the manifold, allowing it to characterize a meaningful geometry across the entire latent space. Using this metric, GAGA can uniformly sample points on the manifold, generate points along geodesics, and interpolate between populations across the learned manifold. GAGA shows competitive performance in simulated and real world datasets, including a 30% improvement over the state-of-the-art methods in single-cell population-level trajectory inference.

Title: Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats

• Authors: Chen Ziwen, Hao Tan, Kai Zhang, Sai Bi, Fujun Luan, Yicong Hong, Li Fuxin, Zexiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.12781
• Pdf URL: https://arxiv.org/pdf/2410.12781
• Copy Paste: [[2410.12781]] Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats(https://arxiv.org/abs/2410.12781)
• Keywords: transformer
• Abstract: We propose Long-LRM, a generalizable 3D Gaussian reconstruction model that is capable of reconstructing a large scene from a long sequence of input images. Specifically, our model can process 32 source images at 960x540 resolution within only 1.3 seconds on a single A100 80G GPU. Our architecture features a mixture of the recent Mamba2 blocks and the classical transformer blocks which allowed many more tokens to be processed than prior work, enhanced by efficient token merging and Gaussian pruning steps that balance between quality and efficiency. Unlike previous feed-forward models that are limited to processing 1~4 input images and can only reconstruct a small portion of a large scene, Long-LRM reconstructs the entire scene in a single feed-forward step. On large-scale scene datasets such as DL3DV-140 and Tanks and Temples, our method achieves performance comparable to optimization-based approaches while being two orders of magnitude more efficient. Project page: this https URL

Title: Context-Scaling versus Task-Scaling in In-Context Learning

• Authors: Amirhesam Abedsoltan, Adityanarayanan Radhakrishnan, Jingfeng Wu, Mikhail Belkin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.12783
• Pdf URL: https://arxiv.org/pdf/2410.12783
• Copy Paste: [[2410.12783]] Context-Scaling versus Task-Scaling in In-Context Learning(https://arxiv.org/abs/2410.12783)
• Keywords: transformer
• Abstract: Transformers exhibit In-Context Learning (ICL), where these models solve new tasks by using examples in the prompt without additional training. In our work, we identify and analyze two key components of ICL: (1) context-scaling, where model performance improves as the number of in-context examples increases and (2) task-scaling, where model performance improves as the number of pre-training tasks increases. While transformers are capable of both context-scaling and task-scaling, we empirically show that standard Multi-Layer Perceptrons (MLPs) with vectorized input are only capable of task-scaling. To understand how transformers are capable of context-scaling, we first propose a significantly simplified transformer architecture without key, query, value weights. We show that it performs ICL comparably to the original GPT-2 model in various statistical learning tasks including linear regression, teacher-student settings. Furthermore, a single block of our simplified transformer can be viewed as data dependent feature map followed by an MLP. This feature map on its own is a powerful predictor that is capable of context-scaling but is not capable of task-scaling. We show empirically that concatenating the output of this feature map with vectorized data as an input to MLPs enables both context-scaling and task-scaling. This finding provides a simple setting to study context and task-scaling for ICL.

Title: Metal Price Spike Prediction via a Neurosymbolic Ensemble Approach

• Authors: Nathaniel Lee, Noel Ngu, Harshdeep Singh Sahdev, Pramod Motaganahall, Al Mehdi Saadat Chowdhury, Bowen Xi, Paulo Shakarian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.12785
• Pdf URL: https://arxiv.org/pdf/2410.12785
• Copy Paste: [[2410.12785]] Metal Price Spike Prediction via a Neurosymbolic Ensemble Approach(https://arxiv.org/abs/2410.12785)
• Keywords: interpretability
• Abstract: Predicting price spikes in critical metals such as Cobalt, Copper, Magnesium, and Nickel is crucial for mitigating economic risks associated with global trends like the energy transition and reshoring of manufacturing. While traditional models have focused on regression-based approaches, our work introduces a neurosymbolic ensemble framework that integrates multiple neural models with symbolic error detection and correction rules. This framework is designed to enhance predictive accuracy by correcting individual model errors and offering interpretability through rule-based explanations. We show that our method provides up to 6.42% improvement in precision, 29.41% increase in recall at 13.24% increase in F1 over the best performing neural models. Further, our method, as it is based on logical rules, has the benefit of affording an explanation as to which combination of neural models directly contribute to a given prediction.

Title: Meta-Chunking: Learning Efficient Text Segmentation via Logical Perception

• Authors: Jihao Zhao, Zhiyuan Ji, Pengnian Qi, Simin Niu, Bo Tang, Feiyu Xiong, Zhiyu Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12788
• Pdf URL: https://arxiv.org/pdf/2410.12788
• Copy Paste: [[2410.12788]] Meta-Chunking: Learning Efficient Text Segmentation via Logical Perception(https://arxiv.org/abs/2410.12788)
• Keywords: large language model, segmentation
• Abstract: Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline, which impacts the quality of knowledge-intensive tasks. This paper introduces the concept of Meta-Chunking, which refers to a granularity between sentences and paragraphs, consisting of a collection of sentences within a paragraph that have deep linguistic logical connections. To implement Meta-Chunking, we designed two strategies based on LLMs: Margin Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform binary classification on whether consecutive sentences need to be segmented, making decisions based on the probability difference obtained from margin sampling. The latter precisely identifies text chunk boundaries by analyzing the characteristics of perplexity distribution. Additionally, considering the inherent complexity of different texts, we propose a strategy that combines Meta-Chunking with dynamic merging to achieve a balance between fine-grained and coarse-grained text chunking. Experiments conducted on eleven datasets demonstrate that Meta-Chunking can more efficiently improve the performance of single-hop and multi-hop question answering based on RAG. For instance, on the 2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only consuming 45.8% of the time. Our code is available at this https URL.

Title: Context is Key(NMF): Modelling Topical Information Dynamics in Chinese Diaspora Media

• Authors: Ross Deans Kristensen-McLachlan, Rebecca M. M. Hicke, Márton Kardos, Mette Thunø
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.12791
• Pdf URL: https://arxiv.org/pdf/2410.12791
• Copy Paste: [[2410.12791]] Context is Key(NMF): Modelling Topical Information Dynamics in Chinese Diaspora Media(https://arxiv.org/abs/2410.12791)
• Keywords: transformer
• Abstract: Does the People's Republic of China (PRC) interfere with European elections through ethnic Chinese diaspora media? This question forms the basis of an ongoing research project exploring how PRC narratives about European elections are represented in Chinese diaspora media, and thus the objectives of PRC news media manipulation. In order to study diaspora media efficiently and at scale, it is necessary to use techniques derived from quantitative text analysis, such as topic modelling. In this paper, we present a pipeline for studying information dynamics in Chinese media. Firstly, we present KeyNMF, a new approach to static and dynamic topic modelling using transformer-based contextual embedding models. We provide benchmark evaluations to demonstrate that our approach is competitive on a number of Chinese datasets and metrics. Secondly, we integrate KeyNMF with existing methods for describing information dynamics in complex systems. We apply this pipeline to data from five news sites, focusing on the period of time leading up to the 2024 European parliamentary elections. Our methods and results demonstrate the effectiveness of KeyNMF for studying information dynamics in Chinese media and lay groundwork for further work addressing the broader research questions.