2024-10-04

Title: Privacy-Preserving SAM Quantization for Efficient Edge Intelligence in Healthcare

• Authors: Zhikai Li, Jing Zhang, Qingyi Gu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01813
• Pdf URL: https://arxiv.org/pdf/2410.01813
• Copy Paste: [[2410.01813]] Privacy-Preserving SAM Quantization for Efficient Edge Intelligence in Healthcare(https://arxiv.org/abs/2410.01813)
• Keywords: secure, security, privacy, protect, attack, data-free, segmentation
• Abstract: The disparity in healthcare personnel expertise and medical resources across different regions of the world is a pressing social issue. Artificial intelligence technology offers new opportunities to alleviate this issue. Segment Anything Model (SAM), which excels in intelligent image segmentation, has demonstrated exceptional performance in medical monitoring and assisted diagnosis. Unfortunately, the huge computational and storage overhead of SAM poses significant challenges for deployment on resource-limited edge devices. Quantization is an effective solution for model compression; however, traditional methods rely heavily on original data for calibration, which raises widespread concerns about medical data privacy and security. In this paper, we propose a data-free quantization framework for SAM, called DFQ-SAM, which learns and calibrates quantization parameters without any original data, thus effectively preserving data privacy during model compression. Specifically, we propose pseudo-positive label evolution for segmentation, combined with patch similarity, to fully leverage the semantic and distribution priors in pre-trained models, which facilitates high-quality data synthesis as a substitute for real data. Furthermore, we introduce scale reparameterization to ensure the accuracy of low-bit quantization. We perform extensive segmentation experiments on various datasets, and DFQ-SAM consistently provides significant performance on low-bit quantization. DFQ-SAM eliminates the need for data transfer in cloud-edge collaboration, thereby protecting sensitive data from potential attacks. It enables secure, fast, and personalized healthcare services at the edge, which enhances system efficiency and optimizes resource allocation, and thus facilitating the pervasive application of artificial intelligence in worldwide healthcare.

Title: Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects

• Authors: Awal Ahmed Fime, Saifuddin Mahmud, Arpita Das, Md. Sunzidul Islam, Hong-Hoon Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01816
• Pdf URL: https://arxiv.org/pdf/2410.01816
• Copy Paste: [[2410.01816]] Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects(https://arxiv.org/abs/2410.01816)
• Keywords: diffusion, transformer, generative
• Abstract: Automatic scene generation is an essential area of research with applications in robotics, recreation, visual representation, training and simulation, education, and more. This survey provides a comprehensive review of the current state-of-the-arts in automatic scene generation, focusing on techniques that leverage machine learning, deep learning, embedded systems, and natural language processing (NLP). We categorize the models into four main types: Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Transformers, and Diffusion Models. Each category is explored in detail, discussing various sub-models and their contributions to the field. We also review the most commonly used datasets, such as COCO-Stuff, Visual Genome, and MS-COCO, which are critical for training and evaluating these models. Methodologies for scene generation are examined, including image-to-3D conversion, text-to-3D generation, UI/layout design, graph-based methods, and interactive scene generation. Evaluation metrics such as Frechet Inception Distance (FID), Kullback-Leibler (KL) Divergence, Inception Score (IS), Intersection over Union (IoU), and Mean Average Precision (mAP) are discussed in the context of their use in assessing model performance. The survey identifies key challenges and limitations in the field, such as maintaining realism, handling complex scenes with multiple objects, and ensuring consistency in object relationships and spatial arrangements. By summarizing recent advances and pinpointing areas for improvement, this survey aims to provide a valuable resource for researchers and practitioners working on automatic scene generation.

Title: From Experts to the Public: Governing Multimodal Language Models in Politically Sensitive Video Analysis

• Authors: Tanusree Sharma, Yujin Potter, Zachary Kilhoffer, Yun Huang, Dawn Song, Yang Wang
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.01817
• Pdf URL: https://arxiv.org/pdf/2410.01817
• Copy Paste: [[2410.01817]] From Experts to the Public: Governing Multimodal Language Models in Politically Sensitive Video Analysis(https://arxiv.org/abs/2410.01817)
• Keywords: large language model
• Abstract: This paper examines the governance of multimodal large language models (MM-LLMs) through individual and collective deliberation, focusing on analyses of politically sensitive videos. We conducted a two-step study: first, interviews with 10 journalists established a baseline understanding of expert video interpretation; second, 114 individuals from the general public engaged in deliberation using this http URL, a platform that facilitates democratic decision-making through decentralized autonomous organization (DAO) mechanisms. Our findings show that while experts emphasized emotion and narrative, the general public prioritized factual clarity, objectivity of the situation, and emotional neutrality. Additionally, we explored the impact of different governance mechanisms: quadratic vs. weighted ranking voting and equal vs. 20-80 power distributions on users decision-making on how AI should behave. Specifically, quadratic voting enhanced perceptions of liberal democracy and political equality, and participants who were more optimistic about AI perceived the voting process to have a higher level of participatory democracy. Our results suggest the potential of applying DAO mechanisms to help democratize AI governance.

Title: PixelBytes: Catching Unified Representation for Multimodal Generation

• Authors: Fabien Furfaro
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01820
• Pdf URL: https://arxiv.org/pdf/2410.01820
• Copy Paste: [[2410.01820]] PixelBytes: Catching Unified Representation for Multimodal Generation(https://arxiv.org/abs/2410.01820)
• Keywords: transformer
• Abstract: This report introduces PixelBytes, a novel approach for unified multimodal representation learning. Inspired by existing sequence models such as Image Transformers, PixelCNN, and Mamba-Bytes, our method aims to capture diverse inputs in a cohesive representation, exploring the integration of different data types, particularly text, audio, and pixelated images (sprites). We conducted experiments on a specialized PixelBytes Pok{é}mon dataset. Initially, we investigated various model architectures, including Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Attention-based models, focusing on bidirectional processing and our convolutional PxBy embedding technique. Subsequently, we evaluated models based on data reduction strategies and the effectiveness of autoregressive learning. We specifically examined Long Short-Term Memory (LSTM) networks in both predictive and autoregressive modes for our main experiments. Our findings suggest that autoregressive models outperform predictive models in this context. By adopting a flexible approach to multimodal modeling, PixelBytes contributes to the ongoing development of foundation models capable of understanding and generating multimodal data. The complete PixelBytes project, including code, models, and datasets, is available online.

Title: Spatial Action Unit Cues for Interpretable Deep Facial Expression Recognition

• Authors: Soufiane Belharbi, Marco Pedersoli, Alessandro Lameiras Koerich, Simon Bacon, Eric Granger
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01848
• Pdf URL: https://arxiv.org/pdf/2410.01848
• Copy Paste: [[2410.01848]] Spatial Action Unit Cues for Interpretable Deep Facial Expression Recognition(https://arxiv.org/abs/2410.01848)
• Keywords: interpretability, transformer
• Abstract: Although state-of-the-art classifiers for facial expression recognition (FER) can achieve a high level of accuracy, they lack interpretability, an important feature for end-users. Experts typically associate spatial action units (AUs) from a codebook to facial regions for the visual interpretation of expressions. In this paper, the same expert steps are followed. A new learning strategy is proposed to explicitly incorporate AU cues into classifier training, allowing to train deep interpretable models. During training, this AU codebook is used, along with the input image expression label, and facial landmarks, to construct a AU heatmap that indicates the most discriminative image regions of interest w.r.t the facial expression. This valuable spatial cue is leveraged to train a deep interpretable classifier for FER. This is achieved by constraining the spatial layer features of a classifier to be correlated with AU heatmaps. Using a composite loss, the classifier is trained to correctly classify an image while yielding interpretable visual layer-wise attention correlated with AU maps, simulating the expert decision process. Our strategy only relies on image class expression for supervision, without additional manual annotations. Our new strategy is generic, and can be applied to any deep CNN- or transformer-based classifier without requiring any architectural change or significant additional training time. Our extensive evaluation on two public benchmarks RAF-DB, and AffectNet datasets shows that our proposed strategy can improve layer-wise interpretability without degrading classification performance. In addition, we explore a common type of interpretable classifiers that rely on class activation mapping (CAM) methods, and show that our approach can also improve CAM interpretability.

Title: Explainable Diagnosis Prediction through Neuro-Symbolic Integration

• Authors: Qiuhao Lu, Rui Li, Elham Sagheb, Andrew Wen, Jinlian Wang, Liwei Wang, Jungwei W. Fan, Hongfang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01855
• Pdf URL: https://arxiv.org/pdf/2410.01855
• Copy Paste: [[2410.01855]] Explainable Diagnosis Prediction through Neuro-Symbolic Integration(https://arxiv.org/abs/2410.01855)
• Keywords: interpretability, explainability
• Abstract: Diagnosis prediction is a critical task in healthcare, where timely and accurate identification of medical conditions can significantly impact patient outcomes. Traditional machine learning and deep learning models have achieved notable success in this domain but often lack interpretability which is a crucial requirement in clinical settings. In this study, we explore the use of neuro-symbolic methods, specifically Logical Neural Networks (LNNs), to develop explainable models for diagnosis prediction. Essentially, we design and implement LNN-based models that integrate domain-specific knowledge through logical rules with learnable thresholds. Our models, particularly $M_{\text{multi-pathway}}$ and $M_{\text{comprehensive}}$, demonstrate superior performance over traditional models such as Logistic Regression, SVM, and Random Forest, achieving higher accuracy (up to 80.52\%) and AUROC scores (up to 0.8457) in the case study of diabetes prediction. The learned weights and thresholds within the LNN models provide direct insights into feature contributions, enhancing interpretability without compromising predictive power. These findings highlight the potential of neuro-symbolic approaches in bridging the gap between accuracy and explainability in healthcare AI applications. By offering transparent and adaptable diagnostic models, our work contributes to the advancement of precision medicine and supports the development of equitable healthcare solutions. Future research will focus on extending these methods to larger and more diverse datasets to further validate their applicability across different medical conditions and populations.

Title: Learning the Optimal Path and DNN Partition for Collaborative Edge Inference

• Authors: Yin Huang, Letian Zhang, Jie Xu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.01857
• Pdf URL: https://arxiv.org/pdf/2410.01857
• Copy Paste: [[2410.01857]] Learning the Optimal Path and DNN Partition for Collaborative Edge Inference(https://arxiv.org/abs/2410.01857)
• Keywords: security
• Abstract: Recent advancements in Deep Neural Networks (DNNs) have catalyzed the development of numerous intelligent mobile applications and services. However, they also introduce significant computational challenges for resource-constrained mobile devices. To address this, collaborative edge inference has been proposed. This method involves partitioning a DNN inference task into several subtasks and distributing these across multiple network nodes. Despite its potential, most current approaches presume known network parameters -- like node processing speeds and link transmission rates -- or rely on a fixed sequence of nodes for processing the DNN subtasks. In this paper, we tackle a more complex scenario where network parameters are unknown and must be learned, and multiple network paths are available for distributing inference tasks. Specifically, we explore the learning problem of selecting the optimal network path and assigning DNN layers to nodes along this path, considering potential security threats and the costs of switching paths. We begin by deriving structural insights from the DNN layer assignment with complete network information, which narrows down the decision space and provides crucial understanding of optimal assignments. We then cast the learning problem with incomplete network information as a novel adversarial group linear bandits problem with switching costs, featuring rewards generation through a combined stochastic and adversarial process. We introduce a new bandit algorithm, B-EXPUCB, which combines elements of the classical blocked EXP3 and LinUCB algorithms, and demonstrate its sublinear regret. Extensive simulations confirm B-EXPUCB's superior performance in learning for collaborative edge inference over existing algorithms.

Title: OCC-MLLM-Alpha:Empowering Multi-modal Large Language Model for the Understanding of Occluded Objects with Self-Supervised Test-Time Learning

• Authors: Shuxin Yang, Xinhan Di
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01861
• Pdf URL: https://arxiv.org/pdf/2410.01861
• Copy Paste: [[2410.01861]] OCC-MLLM-Alpha:Empowering Multi-modal Large Language Model for the Understanding of Occluded Objects with Self-Supervised Test-Time Learning(https://arxiv.org/abs/2410.01861)
• Keywords: large language model
• Abstract: There is a gap in the understanding of occluded objects in existing large-scale visual language multi-modal models. Current state-of-the-art multi-modal models fail to provide satisfactory results in describing occluded objects through universal visual encoders and supervised learning strategies. Therefore, we introduce a multi-modal large language framework and corresponding self-supervised learning strategy with support of 3D generation. We start our experiments comparing with the state-of-the-art models in the evaluation of a large-scale dataset SOMVideo [18]. The initial results demonstrate the improvement of 16.92% in comparison with the state-of-the-art VLM models.

Title: House of Cards: Massive Weights in LLMs

• Authors: Jaehoon Oh, Seungjun Shin, Dokwan Oh
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01866
• Pdf URL: https://arxiv.org/pdf/2410.01866
• Copy Paste: [[2410.01866]] House of Cards: Massive Weights in LLMs(https://arxiv.org/abs/2410.01866)
• Keywords: large language model
• Abstract: Massive activations, which manifest in specific feature dimensions of hidden states, introduce a significant bias in large language models (LLMs), leading to an overemphasis on the corresponding token. In this paper, we identify that massive activations originate not from the hidden state but from the intermediate state of a feed-forward network module in an early layer. Expanding on the previous observation that massive activations occur only in specific feature dimensions, we dive deep into the weights that cause massive activations. Specifically, we define top-$k$ massive weights as the weights that contribute to the dimensions with the top-$k$ magnitudes in the intermediate state. When these massive weights are set to zero, the functionality of LLMs is entirely disrupted. However, when all weights except for massive weights are set to zero, it results in a relatively minor performance drop, even though a much larger number of weights are set to zero. This implies that during the pre-training process, learning is dominantly focused on massive weights. Building on this observation, we propose a simple plug-and-play method called MacDrop (massive weights curriculum dropout), to rely less on massive weights during parameter-efficient fine-tuning. This method applies dropout to the pre-trained massive weights, starting with a high dropout probability and gradually decreasing it as fine-tuning progresses. Through experiments, we demonstrate that MacDrop generally improves performance across zero-shot downstream tasks and generation tasks.

Title: Conformal Prediction Sets Can Cause Disparate Impact

• Authors: Jesse C. Cresswell, Bhargava Kumar, Yi Sui, Mouloud Belbahri
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01888
• Pdf URL: https://arxiv.org/pdf/2410.01888
• Copy Paste: [[2410.01888]] Conformal Prediction Sets Can Cause Disparate Impact(https://arxiv.org/abs/2410.01888)
• Keywords: protect, fair
• Abstract: Although conformal prediction is a promising method for quantifying the uncertainty of machine learning models, the prediction sets it outputs are not inherently actionable. Many applications require a single output to act on, not several. To overcome this, prediction sets can be provided to a human who then makes an informed decision. In any such system it is crucial to ensure the fairness of outcomes across protected groups, and researchers have proposed that Equalized Coverage be used as the standard for fairness. By conducting experiments with human participants, we demonstrate that providing prediction sets can increase the unfairness of their decisions. Disquietingly, we find that providing sets that satisfy Equalized Coverage actually increases unfairness compared to marginal coverage. Instead of equalizing coverage, we propose to equalize set sizes across groups which empirically leads to more fair outcomes.

Title: The potential of LLM-generated reports in DevSecOps

• Authors: Nikolaos Lykousas, Vasileios Argyropoulos, Fran Casino
• Subjects: cs.CR, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.01899
• Pdf URL: https://arxiv.org/pdf/2410.01899
• Copy Paste: [[2410.01899]] The potential of LLM-generated reports in DevSecOps(https://arxiv.org/abs/2410.01899)
• Keywords: security
• Abstract: Alert fatigue is a common issue faced by software teams using the DevSecOps paradigm. The overwhelming number of warnings and alerts generated by security and code scanning tools, particularly in smaller teams where resources are limited, leads to desensitization and diminished responsiveness to security warnings, potentially exposing systems to vulnerabilities. This paper explores the potential of LLMs in generating actionable security reports that emphasize the financial impact and consequences of detected security issues, such as credential leaks, if they remain unaddressed. A survey conducted among developers indicates that LLM-generated reports significantly enhance the likelihood of immediate action on security issues by providing clear, comprehensive, and motivating insights. Integrating these reports into DevSecOps workflows can mitigate attention saturation and alert fatigue, ensuring that critical security warnings are addressed effectively.

Title: Social Media Authentication and Combating Deepfakes using Semi-fragile Invisible Image Watermarking

• Authors: Aakash Varma Nadimpalli, Ajita Rattani
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.01906
• Pdf URL: https://arxiv.org/pdf/2410.01906
• Copy Paste: [[2410.01906]] Social Media Authentication and Combating Deepfakes using Semi-fragile Invisible Image Watermarking(https://arxiv.org/abs/2410.01906)
• Keywords: defense, attack, robust, watermark, generative
• Abstract: With the significant advances in deep generative models for image and video synthesis, Deepfakes and manipulated media have raised severe societal concerns. Conventional machine learning classifiers for deepfake detection often fail to cope with evolving deepfake generation technology and are susceptible to adversarial attacks. Alternatively, invisible image watermarking is being researched as a proactive defense technique that allows media authentication by verifying an invisible secret message embedded in the image pixels. A handful of invisible image watermarking techniques introduced for media authentication have proven vulnerable to basic image processing operations and watermark removal attacks. In response, we have proposed a semi-fragile image watermarking technique that embeds an invisible secret message into real images for media authentication. Our proposed watermarking framework is designed to be fragile to facial manipulations or tampering while being robust to benign image-processing operations and watermark removal attacks. This is facilitated through a unique architecture of our proposed technique consisting of critic and adversarial networks that enforce high image quality and resiliency to watermark removal efforts, respectively, along with the backbone encoder-decoder and the discriminator networks. Thorough experimental investigations on SOTA facial Deepfake datasets demonstrate that our proposed model can embed a $64$-bit secret as an imperceptible image watermark that can be recovered with a high-bit recovery accuracy when benign image processing operations are applied while being non-recoverable when unseen Deepfake manipulations are applied. In addition, our proposed watermarking technique demonstrates high resilience to several white-box and black-box watermark removal attacks. Thus, obtaining state-of-the-art performance.

Title: A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation

• Authors: Liang Chen, Sinan Tan, Zefan Cai, Weichu Xie, Haozhe Zhao, Yichi Zhang, Junyang Lin, Jinze Bai, Tianyu Liu, Baobao Chang
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01912
• Pdf URL: https://arxiv.org/pdf/2410.01912
• Copy Paste: [[2410.01912]] A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation(https://arxiv.org/abs/2410.01912)
• Keywords: diffusion, transformer, generative
• Abstract: This work tackles the information loss bottleneck of vector-quantization (VQ) autoregressive image generation by introducing a novel model architecture called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer predicts more codes for an image by introducing a new autoregression direction, \textit{model depth}, along with the sequence length direction. Compared to traditional 1D autoregression and previous work utilizing similar 2D image decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end model that can generate higher quality images with the same backbone model size and sequence length, opening a new optimization perspective for autoregressive image generation. Furthermore, our experiments reveal that the DnD-Transformer's potential extends beyond generating natural images. It can even generate images with rich text and graphical elements in a self-supervised manner, demonstrating an understanding of these combined modalities. This has not been previously demonstrated for popular vision generative models such as diffusion models, showing a spark of vision-language intelligence when trained solely on images. Code, datasets and models are open at this https URL.

Title: Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo

• Authors: Shengyu Feng, Xiang Kong, Shuang Ma, Aonan Zhang, Dong Yin, Chong Wang, Ruoming Pang, Yiming Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01920
• Pdf URL: https://arxiv.org/pdf/2410.01920
• Copy Paste: [[2410.01920]] Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo(https://arxiv.org/abs/2410.01920)
• Keywords: large language model
• Abstract: Augmenting the multi-step reasoning abilities of Large Language Models (LLMs) has been a persistent challenge. Recently, verification has shown promise in improving solution consistency by evaluating generated outputs. However, current verification approaches suffer from sampling inefficiencies, requiring a large number of samples to achieve satisfactory performance. Additionally, training an effective verifier often depends on extensive process supervision, which is costly to acquire. In this paper, we address these limitations by introducing a novel verification method based on Twisted Sequential Monte Carlo (TSMC). TSMC sequentially refines its sampling effort to focus exploration on promising candidates, resulting in more efficient generation of high-quality solutions. We apply TSMC to LLMs by estimating the expected future rewards at partial solutions. This approach results in a more straightforward training target that eliminates the need for step-wise human annotations. We empirically demonstrate the advantages of our method across multiple math benchmarks, and also validate our theoretical analysis of both our approach and existing verification methods.

Title: NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel

• Authors: Gabriel Thompson, Kai Yue, Chau-Wai Wong, Huaiyu Dai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01922
• Pdf URL: https://arxiv.org/pdf/2410.01922
• Copy Paste: [[2410.01922]] NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel(https://arxiv.org/abs/2410.01922)
• Keywords: robust, federate
• Abstract: Decentralized federated learning (DFL) is a collaborative machine learning framework for training a model across participants without a central server or raw data exchange. DFL faces challenges due to statistical heterogeneity, as participants often possess different data distributions reflecting local environments and user behaviors. Recent work has shown that the neural tangent kernel (NTK) approach, when applied to federated learning in a centralized framework, can lead to improved performance. The NTK-based update mechanism is more expressive than typical gradient descent methods, enabling more efficient convergence and better handling of data heterogeneity. We propose an approach leveraging the NTK to train client models in the decentralized setting, while introducing a synergy between NTK-based evolution and model averaging. This synergy exploits inter-model variance and improves both accuracy and convergence in heterogeneous settings. Our model averaging technique significantly enhances performance, boosting accuracy by at least 10% compared to the mean local model accuracy. Empirical results demonstrate that our approach consistently achieves higher accuracy than baselines in highly heterogeneous settings, where other approaches often underperform. Additionally, it reaches target performance in 4.6 times fewer communication rounds. We validate our approach across multiple datasets, network topologies, and heterogeneity settings to ensure robustness and generalizability.

Title: MARPLE: A Benchmark for Long-Horizon Inference

• Authors: Emily Jin, Zhuoyi Huang, Jan-Philipp Fränken, Weiyu Liu, Hannah Cha, Erik Brockbank, Sarah Wu, Ruohan Zhang, Jiajun Wu, Tobias Gerstenberg
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01926
• Pdf URL: https://arxiv.org/pdf/2410.01926
• Copy Paste: [[2410.01926]] MARPLE: A Benchmark for Long-Horizon Inference(https://arxiv.org/abs/2410.01926)
• Keywords: robust
• Abstract: Reconstructing past events requires reasoning across long time horizons. To figure out what happened, we need to use our prior knowledge about the world and human behavior and draw inferences from various sources of evidence including visual, language, and auditory cues. We introduce MARPLE, a benchmark for evaluating long-horizon inference capabilities using multi-modal evidence. Our benchmark features agents interacting with simulated households, supporting vision, language, and auditory stimuli, as well as procedurally generated environments and agent behaviors. Inspired by classic ``whodunit'' stories, we ask AI models and human participants to infer which agent caused a change in the environment based on a step-by-step replay of what actually happened. The goal is to correctly identify the culprit as early as possible. Our findings show that human participants outperform both traditional Monte Carlo simulation methods and an LLM baseline (GPT-4) on this task. Compared to humans, traditional inference models are less robust and performant, while GPT-4 has difficulty comprehending environmental changes. We analyze what factors influence inference performance and ablate different modes of evidence, finding that all modes are valuable for performance. Overall, our experiments demonstrate that the long-horizon, multimodal inference tasks in our benchmark present a challenge to current models.

Title: Deep learning assisted high resolution microscopy image processing for phase segmentation in functional composite materials

• Authors: Ganesh Raghavendran (1), Bing Han (1), Fortune Adekogbe (4), Shuang Bai (2), Bingyu Lu (1), William Wu (5), Minghao Zhang (1), Ying Shirley Meng (1 and 3) ((1) Department of NanoEngineering-University of California San Diego, (2) Department of NanoEngineering-University of California San Diego (3) Pritzker School of Molecular Engineering-University of Chicago, (4) Department of Chemical and Petroleum Engineering-University of Lagos, (5) Del Norte High School)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01928
• Pdf URL: https://arxiv.org/pdf/2410.01928
• Copy Paste: [[2410.01928]] Deep learning assisted high resolution microscopy image processing for phase segmentation in functional composite materials(https://arxiv.org/abs/2410.01928)
• Keywords: segmentation
• Abstract: In the domain of battery research, the processing of high-resolution microscopy images is a challenging task, as it involves dealing with complex images and requires a prior understanding of the components involved. The utilization of deep learning methodologies for image analysis has attracted considerable interest in recent years, with multiple investigations employing such techniques for image segmentation and analysis within the realm of battery research. However, the automated analysis of high-resolution microscopy images for detecting phases and components in composite materials is still an underexplored area. This work proposes a novel workflow for detecting components and phase segmentation from raw high resolution transmission electron microscopy (TEM) images using a trained U-Net segmentation model. The developed model can expedite the detection of components and phase segmentation, diminishing the temporal and cognitive demands associated with scrutinizing an extensive array of TEM images, thereby mitigating the potential for human errors. This approach presents a novel and efficient image analysis approach with broad applicability beyond the battery field and holds potential for application in other related domains characterized by phase and composition distribution, such as alloy production.

Title: TAEGAN: Generating Synthetic Tabular Data For Data Augmentation

• Authors: Jiayu Li, Zilong Zhao, Kevin Yee, Uzair Javaid, Biplab Sikdar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01933
• Pdf URL: https://arxiv.org/pdf/2410.01933
• Copy Paste: [[2410.01933]] TAEGAN: Generating Synthetic Tabular Data For Data Augmentation(https://arxiv.org/abs/2410.01933)
• Keywords: privacy, generative, large language model
• Abstract: Synthetic tabular data generation has gained significant attention for its potential in data augmentation, software testing and privacy-preserving data sharing. However, most research has primarily focused on larger datasets and evaluating their quality in terms of metrics like column-wise statistical distributions and inter-feature correlations, while often overlooking its utility for data augmentation, particularly for datasets whose data is scarce. In this paper, we propose Tabular Auto-Encoder Generative Adversarial Network (TAEGAN), an improved GAN-based framework for generating high-quality tabular data. Although large language models (LLMs)-based methods represent the state-of-the-art in synthetic tabular data generation, they are often overkill for small datasets due to their extensive size and complexity. TAEGAN employs a masked auto-encoder as the generator, which for the first time introduces the power of self-supervised pre-training in tabular data generation so that essentially exposes the networks to more information. We extensively evaluate TAEGAN against five state-of-the-art synthetic tabular data generation algorithms. Results from 10 datasets show that TAEGAN outperforms existing deep-learning-based tabular data generation models on 9 out of 10 datasets on the machine learning efficacy and achieves superior data augmentation performance on 7 out of 8 smaller datasets.

Title: CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL

• Authors: Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, Sercan O. Arik
• Subjects: cs.LG, cs.AI, cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2410.01943
• Pdf URL: https://arxiv.org/pdf/2410.01943
• Copy Paste: [[2410.01943]] CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL(https://arxiv.org/abs/2410.01943)
• Keywords: robust, large language model
• Abstract: In tackling the challenges of large language model (LLM) performance for Text-to-SQL tasks, we introduce CHASE-SQL, a new framework that employs innovative strategies, using test-time compute in multi-agent modeling to improve candidate generation and selection. CHASE-SQL leverages LLMs' intrinsic knowledge to generate diverse and high-quality SQL candidates using different LLM generators with: (1) a divide-and-conquer method that decomposes complex queries into manageable sub-queries in a single LLM call; (2) chain-of-thought reasoning based on query execution plans, reflecting the steps a database engine takes during execution; and (3) a unique instance-aware synthetic example generation technique, which offers specific few-shot demonstrations tailored to test this http URL identify the best candidate, a selection agent is employed to rank the candidates through pairwise comparisons with a fine-tuned binary-candidates selection LLM. This selection approach has been demonstrated to be more robust over alternatives. The proposed generators-selector framework not only enhances the quality and diversity of SQL queries but also outperforms previous methods. Overall, our proposed CHASE-SQL achieves the state-of-the-art execution accuracy of 73.0% and 73.01% on the test set and development set of the notable BIRD Text-to-SQL dataset benchmark, rendering CHASE-SQL the top submission of the leaderboard (at the time of paper submission).

Title: One-step Noisy Label Mitigation

• Authors: Hao Li, Jiayang Gu, Jingkuan Song, An Zhang, Lianli Gao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01944
• Pdf URL: https://arxiv.org/pdf/2410.01944
• Copy Paste: [[2410.01944]] One-step Noisy Label Mitigation(https://arxiv.org/abs/2410.01944)
• Keywords: robust
• Abstract: Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical, as obtaining entirely clean or human-annotated samples for large-scale pre-training tasks is often impractical. Nonetheless, existing noise mitigation methods often encounter limitations in practical applications due to their task-specific design, model dependency, and significant computational overhead. In this work, we exploit the properties of high-dimensional orthogonality to identify a robust and effective boundary in cone space for separating clean and noisy samples. Building on this, we propose One-step Anti-Noise (OSA), a model-agnostic noisy label mitigation paradigm that employs an estimator model and a scoring function to assess the noise level of input pairs through just one-step inference, a cost-efficient process. We empirically demonstrate the superiority of OSA, highlighting its enhanced training robustness, improved task transferability, ease of deployment, and reduced computational costs across various benchmarks, models, and tasks. Our code is released at this https URL.

Title: Differentially Private Parameter-Efficient Fine-tuning for Large ASR Models

• Authors: Hongbin Liu, Lun Wang, Om Thakkar, Abhradeep Thakurta, Arun Narayanan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.01948
• Pdf URL: https://arxiv.org/pdf/2410.01948
• Copy Paste: [[2410.01948]] Differentially Private Parameter-Efficient Fine-tuning for Large ASR Models(https://arxiv.org/abs/2410.01948)
• Keywords: privacy
• Abstract: Large ASR models can inadvertently leak sensitive information, which can be mitigated by formal privacy measures like differential privacy (DP). However, traditional DP training is computationally expensive, and can hurt model performance. Our study explores DP parameter-efficient fine-tuning as a way to mitigate privacy risks with smaller computation and performance costs for ASR models. Through extensive experimentation and progressive optimization, we achieve 4.6%/8.1% word error rate on LibriSpeech clean/other test-sets, setting a new performance benchmark while maintaining (10, 3.52e-6)-DP in fine-tuning a large ASR model with over 600M parameters.

Title: Discrete Copula Diffusion

• Authors: Anji Liu, Oliver Broadrick, Mathias Niepert, Guy Van den Broeck
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01949
• Pdf URL: https://arxiv.org/pdf/2410.01949
• Copy Paste: [[2410.01949]] Discrete Copula Diffusion(https://arxiv.org/abs/2410.01949)
• Keywords: diffusion, generative
• Abstract: Discrete diffusion models have recently shown significant progress in modeling complex data, such as natural languages and DNA sequences. However, unlike diffusion models for continuous data, which can generate high-quality samples in just a few denoising steps, modern discrete diffusion models still require hundreds or even thousands of denoising steps to perform well. In this paper, we identify a fundamental limitation that prevents discrete diffusion models from achieving strong performance with fewer steps -- they fail to capture dependencies between output variables at each denoising step. To address this issue, we provide a formal explanation and introduce a general approach to supplement the missing dependency information by incorporating another deep generative model, termed the copula model. Our method does not require fine-tuning either the diffusion model or the copula model, yet it enables high-quality sample generation with significantly fewer denoising steps. When we apply this approach to autoregressive copula models, the combined model outperforms both models individually in unconditional and conditional text generation. Specifically, the hybrid model achieves better (un)conditional text generation using 8 to 32 times fewer denoising steps than the diffusion model alone. In addition to presenting an effective discrete diffusion generation algorithm, this paper emphasizes the importance of modeling inter-variable dependencies in discrete diffusion.

Title: Score-based pullback Riemannian geometry

• Authors: Willem Diepeveen, Georgios Batzolis, Zakhar Shumaylov, Carola-Bibiane Schönlieb
• Subjects: cs.LG, math.DG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01950
• Pdf URL: https://arxiv.org/pdf/2410.01950
• Copy Paste: [[2410.01950]] Score-based pullback Riemannian geometry(https://arxiv.org/abs/2410.01950)
• Keywords: generative
• Abstract: Data-driven Riemannian geometry has emerged as a powerful tool for interpretable representation learning, offering improved efficiency in downstream tasks. Moving forward, it is crucial to balance cheap manifold mappings with efficient training algorithms. In this work, we integrate concepts from pullback Riemannian geometry and generative models to propose a framework for data-driven Riemannian geometry that is scalable in both geometry and learning: score-based pullback Riemannian geometry. Focusing on unimodal distributions as a first step, we propose a score-based Riemannian structure with closed-form geodesics that pass through the data probability density. With this structure, we construct a Riemannian autoencoder (RAE) with error bounds for discovering the correct data manifold dimension. This framework can naturally be used with anisotropic normalizing flows by adopting isometry regularization during training. Through numerical experiments on various datasets, we demonstrate that our framework not only produces high-quality geodesics through the data support, but also reliably estimates the intrinsic dimension of the data manifold and provides a global chart of the manifold, even in high-dimensional ambient spaces.

Title: TypedThinker: Typed Thinking Improves Large Language Model Reasoning

• Authors: Danqing Wang, Jianxin Ma, Fei Fang, Lei Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01952
• Pdf URL: https://arxiv.org/pdf/2410.01952
• Copy Paste: [[2410.01952]] TypedThinker: Typed Thinking Improves Large Language Model Reasoning(https://arxiv.org/abs/2410.01952)
• Keywords: large language model
• Abstract: Despite significant advancements in the reasoning capabilities of Large Language Models (LLMs), the lack of diverse reasoning solutions often makes them trapped in a limited solution search area. In this paper, we propose TypedThinker, a novel framework that enhances LLMs' problem-solving abilities by incorporating multiple reasoning types (deductive, inductive, abductive, and analogical). Our analysis across four benchmarks reveals that different reasoning types uniquely solve distinct sets of problems, highlighting the importance of diverse thinking approaches. TypedThinker addresses two key challenges: selecting appropriate reasoning types for given problems and effectively implementing specific reasoning types. Through self-training on successful experiences, TypedThinker learns an implicit policy for reasoning type selection and application. Experimental results demonstrate significant improvements over baseline models, with accuracy increases of 3.4% for Mistral 7B and 16.7% for LLaMA3 8B across four reasoning benchmarks. Notably, TypedThinker shows effective generalization to new benchmarks and can further enhance the reasoning capability of powerful models like GPT-4o. The code is released at this https URL.

Title: Generate then Refine: Data Augmentation for Zero-shot Intent Detection

• Authors: I-Fan Lin, Faegheh Hasibi, Suzan Verberne
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01953
• Pdf URL: https://arxiv.org/pdf/2410.01953
• Copy Paste: [[2410.01953]] Generate then Refine: Data Augmentation for Zero-shot Intent Detection(https://arxiv.org/abs/2410.01953)
• Keywords: generative, large language model
• Abstract: In this short paper we propose a data augmentation method for intent detection in zero-resource domains. Existing data augmentation methods rely on few labelled examples for each intent category, which can be expensive in settings with many possible intents. We use a two-stage approach: First, we generate utterances for intent labels using an open-source large language model in a zero-shot setting. Second, we develop a smaller sequence-to-sequence model (the Refiner), to improve the generated utterances. The Refiner is fine-tuned on seen domains and then applied to unseen domains. We evaluate our method by training an intent classifier on the generated data, and evaluating it on real (human) data. We find that the Refiner significantly improves the data utility and diversity over the zero-shot LLM baseline for unseen domains and over common baseline approaches. Our results indicate that a two-step approach of a generative LLM in zero-shot setting and a smaller sequence-to-sequence model can provide high-quality data for intent detection.

Title: How Reliable Is Human Feedback For Aligning Large Language Models?

• Authors: Min-Hsuan Yeh, Leitian Tao, Jeffrey Wang, Xuefeng Du, Yixuan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01957
• Pdf URL: https://arxiv.org/pdf/2410.01957
• Copy Paste: [[2410.01957]] How Reliable Is Human Feedback For Aligning Large Language Models?(https://arxiv.org/abs/2410.01957)
• Keywords: large language model
• Abstract: Most alignment research today focuses on designing new learning algorithms using datasets like Anthropic-HH, assuming human feedback data is inherently reliable. However, little attention has been given to the qualitative unreliability of human feedback and its impact on alignment. To address this gap, we conduct a comprehensive study and provide an in-depth analysis of human feedback data. We assess feedback reliability using a committee of gold reward models, revealing that over 25% of the dataset shows low or no agreement with these models, implying a high degree of unreliability. Through a qualitative analysis, we identify six key sources of unreliability, such as mis-labeling, subjective preferences, differing criteria and thresholds for helpfulness and harmlessness, etc. Lastly, to mitigate unreliability, we propose Source-Aware Cleaning, an automatic data-cleaning method guided by the insight of our qualitative analysis, to significantly improve data quality. Extensive experiments demonstrate that models trained on our cleaned dataset, HH-Clean, substantially outperform those trained on the original dataset. We release HH-Clean to support more reliable LLM alignment evaluation in the future.

Title: Language Supervised Human Action Recognition with Salient Fusion: Construction Worker Action Recognition as a Use Case

• Authors: Mohammad Mahdavian, Mohammad Loni, Mo Chen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.01962
• Pdf URL: https://arxiv.org/pdf/2410.01962
• Copy Paste: [[2410.01962]] Language Supervised Human Action Recognition with Salient Fusion: Construction Worker Action Recognition as a Use Case(https://arxiv.org/abs/2410.01962)
• Keywords: robust, extraction, transformer
• Abstract: Detecting human actions is a crucial task for autonomous robots and vehicles, often requiring the integration of various data modalities for improved accuracy. In this study, we introduce a novel approach to Human Action Recognition (HAR) based on skeleton and visual cues. Our method leverages a language model to guide the feature extraction process in the skeleton encoder. Specifically, we employ learnable prompts for the language model conditioned on the skeleton modality to optimize feature representation. Furthermore, we propose a fusion mechanism that combines dual-modality features using a salient fusion module, incorporating attention and transformer mechanisms to address the modalities' high dimensionality. This fusion process prioritizes informative video frames and body joints, enhancing the recognition accuracy of human actions. Additionally, we introduce a new dataset tailored for real-world robotic applications in construction sites, featuring visual, skeleton, and depth data modalities, named VolvoConstAct. This dataset serves to facilitate the training and evaluation of machine learning models to instruct autonomous construction machines for performing necessary tasks in the real world construction zones. To evaluate our approach, we conduct experiments on our dataset as well as three widely used public datasets, NTU-RGB+D, NTU-RGB+D120 and NW-UCLA. Results reveal that our proposed method achieves promising performance across all datasets, demonstrating its robustness and potential for various applications. The codes and dataset are available at: this https URL

Title: UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription

• Authors: Reza Basiri, Ali Abedi, Chau Nguyen, Milos R. Popovic, Shehroz S. Khan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01989
• Pdf URL: https://arxiv.org/pdf/2410.01989
• Copy Paste: [[2410.01989]] UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription(https://arxiv.org/abs/2410.01989)
• Keywords: transformer, generative, large language model
• Abstract: Diabetic foot ulcers (DFUs) are a leading cause of hospitalizations and lower limb amputations, placing a substantial burden on patients and healthcare systems. Early detection and accurate classification of DFUs are critical for preventing serious complications, yet many patients experience delays in receiving care due to limited access to specialized services. Telehealth has emerged as a promising solution, improving access to care and reducing the need for in-person visits. The integration of artificial intelligence and pattern recognition into telemedicine has further enhanced DFU management by enabling automatic detection, classification, and monitoring from images. Despite advancements in artificial intelligence-driven approaches for DFU image analysis, the application of large language models for DFU image transcription has not yet been explored. To address this gap, we introduce UlcerGPT, a novel multimodal approach leveraging large language and vision models for DFU image transcription. This framework combines advanced vision and language models, such as Large Language and Vision Assistant and Chat Generative Pre-trained Transformer, to transcribe DFU images by jointly detecting, classifying, and localizing regions of interest. Through detailed experiments on a public dataset, evaluated by expert clinicians, UlcerGPT demonstrates promising results in the accuracy and efficiency of DFU transcription, offering potential support for clinicians in delivering timely care via telemedicine.

Title: FairlyUncertain: A Comprehensive Benchmark of Uncertainty in Algorithmic Fairness

• Authors: Lucas Rosenblatt, R. Teal Witter
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02005
• Pdf URL: https://arxiv.org/pdf/2410.02005
• Copy Paste: [[2410.02005]] FairlyUncertain: A Comprehensive Benchmark of Uncertainty in Algorithmic Fairness(https://arxiv.org/abs/2410.02005)
• Keywords: fair
• Abstract: Fair predictive algorithms hinge on both equality and trust, yet inherent uncertainty in real-world data challenges our ability to make consistent, fair, and calibrated decisions. While fairly managing predictive error has been extensively explored, some recent work has begun to address the challenge of fairly accounting for irreducible prediction uncertainty. However, a clear taxonomy and well-specified objectives for integrating uncertainty into fairness remains undefined. We address this gap by introducing FairlyUncertain, an axiomatic benchmark for evaluating uncertainty estimates in fairness. Our benchmark posits that fair predictive uncertainty estimates should be consistent across learning pipelines and calibrated to observed randomness. Through extensive experiments on ten popular fairness datasets, our evaluation reveals: (1) A theoretically justified and simple method for estimating uncertainty in binary settings is more consistent and calibrated than prior work; (2) Abstaining from binary predictions, even with improved uncertainty estimates, reduces error but does not alleviate outcome imbalances between demographic groups; (3) Incorporating consistent and calibrated uncertainty estimates in regression tasks improves fairness without any explicit fairness interventions. Additionally, our benchmark package is designed to be extensible and open-source, to grow with the field. By providing a standardized framework for assessing the interplay between uncertainty and fairness, FairlyUncertain paves the way for more equitable and trustworthy machine learning practices.

Title: Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration

• Authors: Vasilis Siomos, Sergio Naval-Marimont, Jonathan Passerat-Palmbach, Giacomo Tarroni
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02006
• Pdf URL: https://arxiv.org/pdf/2410.02006
• Copy Paste: [[2410.02006]] Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration(https://arxiv.org/abs/2410.02006)
• Keywords: privacy, robust, federate
• Abstract: Federated learning is a decentralized collaborative training paradigm that preserves stakeholders' data ownership while improving performance and generalization. However, statistical heterogeneity among client datasets poses a fundamental challenge by degrading system performance. To address this issue, we propose Adaptive Normalization-free Feature Recalibration (ANFR), an architecture-level approach that combines weight standardization and channel attention. Weight standardization normalizes the weights of layers instead of activations. This is less susceptible to mismatched client statistics and inconsistent averaging, thereby more robust under heterogeneity. Channel attention produces learnable scaling factors for feature maps, suppressing those that are inconsistent between clients due to heterogeneity. We demonstrate that combining these techniques boosts model performance beyond their individual contributions, by enhancing class selectivity and optimizing channel attention weight distribution. ANFR operates independently of the aggregation method and is effective in both global and personalized federated learning settings, with minimal computational overhead. Furthermore, when training with differential privacy, ANFR achieves an appealing balance between privacy and utility, enabling strong privacy guarantees without sacrificing performance. By integrating weight standardization and channel attention in the backbone model, ANFR offers a novel and versatile approach to the challenge of statistical heterogeneity. We demonstrate through extensive experiments that ANFR consistently outperforms established baselines across various aggregation methods, datasets, and heterogeneity conditions.

Title: Adaptively Private Next-Token Prediction of Large Language Models

• Authors: James Flemings, Meisam Razaviyayn, Murali Annavaram
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02016
• Pdf URL: https://arxiv.org/pdf/2410.02016
• Copy Paste: [[2410.02016]] Adaptively Private Next-Token Prediction of Large Language Models(https://arxiv.org/abs/2410.02016)
• Keywords: privacy, protect, large language model
• Abstract: As Large Language Models (LLMs) proliferate, developing privacy safeguards for these models is crucial. One popular safeguard involves training LLMs in a differentially private manner. However, such solutions are shown to be computationally expensive and detrimental to the utility of these models. Since LLMs are deployed on the cloud and thus only accessible via an API, a Machine Learning as a Service (MLaaS) provider can protect its downstream data by privatizing the predictions during the decoding process. However, the practicality of such solutions still largely lags behind DP training methods. One recent promising approach, Private Mixing of Ensemble Distributions (PMixED), avoids additive noise by sampling from the output distributions of private LLMs mixed with the output distribution of a public model. Yet, PMixED must satisfy a fixed privacy level for a given number of queries, which is difficult for an analyst to estimate before inference and, hence, does not scale. To this end, we relax the requirements to a more practical setting by introducing Adaptive PMixED (AdaPMixED), a private decoding framework based on PMixED that is adaptive to the private and public output distributions evaluated on a given input query. In this setting, we introduce a noisy screening mechanism that filters out queries with potentially expensive privacy loss, and a data-dependent analysis that exploits the divergence of the private and public output distributions in its privacy loss calculation. Our experimental evaluations demonstrate that our mechanism and analysis can reduce the privacy loss by 16x while preserving the utility over the original PMixED. Furthermore, performing 100K predictions with AdaPMixED still achieves strong utility and a reasonable data-dependent privacy loss of 5.25.

Title: DeepProtein: Deep Learning Library and Benchmark for Protein Sequence Learning

• Authors: Jiaqing Xie, Yue Zhao, Tianfan Fu
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.02023
• Pdf URL: https://arxiv.org/pdf/2410.02023
• Copy Paste: [[2410.02023]] DeepProtein: Deep Learning Library and Benchmark for Protein Sequence Learning(https://arxiv.org/abs/2410.02023)
• Keywords: transformer
• Abstract: In recent years, deep learning has revolutionized the field of protein science, enabling advancements in predicting protein properties, structural folding and interactions. This paper presents DeepProtein, a comprehensive and user-friendly deep learning library specifically designed for protein-related tasks. DeepProtein integrates a couple of state-of-the-art neural network architectures, which include convolutional neural network (CNN), recurrent neural network (RNN), transformer, graph neural network (GNN), and graph transformer (GT). It provides user-friendly interfaces, facilitating domain researchers in applying deep learning techniques to protein data. Also, we curate a benchmark that evaluates these neural architectures on a variety of protein tasks, including protein function prediction, protein localization prediction, and protein-protein interaction prediction, showcasing its superior performance and scalability. Additionally, we provide detailed documentation and tutorials to promote accessibility and encourage reproducible research. This library is extended from a well-known drug discovery library, DeepPurpose and publicly available at this https URL.

Title: Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions

• Authors: Qian Ruan, Ilia Kuznetsov, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02028
• Pdf URL: https://arxiv.org/pdf/2410.02028
• Copy Paste: [[2410.02028]] Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions(https://arxiv.org/abs/2410.02028)
• Keywords: large language model
• Abstract: Classification is a core NLP task architecture with many potential applications. While large language models (LLMs) have brought substantial advancements in text generation, their potential for enhancing classification tasks remains underexplored. To address this gap, we propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task. Our extensive experiments and systematic comparisons with various training approaches and a representative selection of LLMs yield new insights into their application for EIC. We investigate the generalizability of these findings on five further classification tasks. To demonstrate the proposed methods and address the data shortage for empirical edit analysis, we use our best-performing EIC model to create Re3-Sci2.0, a new large-scale dataset of 1,780 scientific document revisions with over 94k labeled edits. The quality of the dataset is assessed through human evaluation. The new dataset enables an in-depth empirical study of human editing behavior in academic writing. We make our experimental framework, models and data publicly available.

Title: XChainWatcher: Monitoring and Identifying Attacks in Cross-Chain Bridges

• Authors: André Augusto, Rafael Belchior, Jonas Pfannschmidt, André Vasconcelos, Miguel Correia
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.02029
• Pdf URL: https://arxiv.org/pdf/2410.02029
• Copy Paste: [[2410.02029]] XChainWatcher: Monitoring and Identifying Attacks in Cross-Chain Bridges(https://arxiv.org/abs/2410.02029)
• Keywords: protect, attack
• Abstract: Cross-chain bridges are widely used blockchain interoperability mechanisms. However, several of these bridges have vulnerabilities that have caused 3.2 billion dollars in losses since May 2021. Some studies have revealed the existence of these vulnerabilities, but little quantitative research is available, and there are no safeguard mechanisms to protect bridges from such attacks. We propose XChainWatcher, the first mechanism for monitoring bridges and detecting attacks against them. XChainWatcher relies on a cross-chain model powered by a Datalog engine, designed to be pluggable into any cross-chain bridge. Analyzing data from the Ronin and Nomad bridges, we successfully identified the transactions that led to losses of \$611M and \$190M USD, respectively. XChainWatcher not only uncovers successful attacks but also reveals unintended behavior, such as 37 cross-chain transactions (cctx) that these bridges should not have accepted, failed attempts to exploit Nomad, over \$7.8M locked on one chain but never released on Ethereum, and \$200K lost due to inadequate interaction with bridges. We provide the first open-source dataset of 81,000 cctxs across three blockchains, capturing \$585M and \$3.7B in token transfers in Nomad and Ronin, respectively.

Title: EAB-FL: Exacerbating Algorithmic Bias through Model Poisoning Attacks in Federated Learning

• Authors: Syed Irfan Ali Meerza, Jian Liu
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02042
• Pdf URL: https://arxiv.org/pdf/2410.02042
• Copy Paste: [[2410.02042]] EAB-FL: Exacerbating Algorithmic Bias through Model Poisoning Attacks in Federated Learning(https://arxiv.org/abs/2410.02042)
• Keywords: secure, privacy, attack, federate, fair
• Abstract: Federated Learning (FL) is a technique that allows multiple parties to train a shared model collaboratively without disclosing their private data. It has become increasingly popular due to its distinct privacy advantages. However, FL models can suffer from biases against certain demographic groups (e.g., racial and gender groups) due to the heterogeneity of data and party selection. Researchers have proposed various strategies for characterizing the group fairness of FL algorithms to address this issue. However, the effectiveness of these strategies in the face of deliberate adversarial attacks has not been fully explored. Although existing studies have revealed various threats (e.g., model poisoning attacks) against FL systems caused by malicious participants, their primary aim is to decrease model accuracy, while the potential of leveraging poisonous model updates to exacerbate model unfairness remains unexplored. In this paper, we propose a new type of model poisoning attack, EAB-FL, with a focus on exacerbating group unfairness while maintaining a good level of model utility. Extensive experiments on three datasets demonstrate the effectiveness and efficiency of our attack, even with state-of-the-art fairness optimization algorithms and secure aggregation rules employed.

Title: Impact of White-Box Adversarial Attacks on Convolutional Neural Networks

• Authors: Rakesh Podder, Sudipto Ghosh
• Subjects: cs.CR, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2410.02043
• Pdf URL: https://arxiv.org/pdf/2410.02043
• Copy Paste: [[2410.02043]] Impact of White-Box Adversarial Attacks on Convolutional Neural Networks(https://arxiv.org/abs/2410.02043)
• Keywords: security, protect, defense, attack, robust
• Abstract: Autonomous vehicle navigation and healthcare diagnostics are among the many fields where the reliability and security of machine learning models for image data are critical. We conduct a comprehensive investigation into the susceptibility of Convolutional Neural Networks (CNNs), which are widely used for image data, to white-box adversarial attacks. We investigate the effects of various sophisticated attacks -- Fast Gradient Sign Method, Basic Iterative Method, Jacobian-based Saliency Map Attack, Carlini & Wagner, Projected Gradient Descent, and DeepFool -- on CNN performance metrics, (e.g., loss, accuracy), the differential efficacy of adversarial techniques in increasing error rates, the relationship between perceived image quality metrics (e.g., ERGAS, PSNR, SSIM, and SAM) and classification performance, and the comparative effectiveness of iterative versus single-step attacks. Using the MNIST, CIFAR-10, CIFAR-100, and Fashio_MNIST datasets, we explore the effect of different attacks on the CNNs performance metrics by varying the hyperparameters of CNNs. Our study provides insights into the robustness of CNNs against adversarial threats, pinpoints vulnerabilities, and underscores the urgent need for developing robust defense mechanisms to protect CNNs and ensuring their trustworthy deployment in real-world scenarios.

Title: Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description

• Authors: Mahshid Dehghani, Amirahmad Shafiee, Ali Shafiei, Neda Fallah, Farahmand Alizadeh, Mohammad Mehdi Gholinejad, Hamid Behroozi, Jafar Habibi, Ehsaneddin Asgari
• Subjects: cs.CV, cs.CL, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.02049
• Pdf URL: https://arxiv.org/pdf/2410.02049
• Copy Paste: [[2410.02049]] Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description(https://arxiv.org/abs/2410.02049)
• Keywords: large language model
• Abstract: Existing 3D facial emotion modeling have been constrained by limited emotion classes and insufficient datasets. This paper introduces "Emo3D", an extensive "Text-Image-Expression dataset" spanning a wide spectrum of human emotions, each paired with images and 3D blendshapes. Leveraging Large Language Models (LLMs), we generate a diverse array of textual descriptions, facilitating the capture of a broad spectrum of emotional expressions. Using this unique dataset, we conduct a comprehensive evaluation of language-based models' fine-tuning and vision-language models like Contranstive Language Image Pretraining (CLIP) for 3D facial expression synthesis. We also introduce a new evaluation metric for this task to more directly measure the conveyed emotion. Our new evaluation metric, Emo3D, demonstrates its superiority over Mean Squared Error (MSE) metrics in assessing visual-text alignment and semantic richness in 3D facial expressions associated with human emotions. "Emo3D" has great applications in animation design, virtual reality, and emotional human-computer interaction.

Title: Using Style Ambiguity Loss to Improve Aesthetics of Diffusion Models

• Authors: James Baker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02055
• Pdf URL: https://arxiv.org/pdf/2410.02055
• Copy Paste: [[2410.02055]] Using Style Ambiguity Loss to Improve Aesthetics of Diffusion Models(https://arxiv.org/abs/2410.02055)
• Keywords: diffusion
• Abstract: Teaching text-to-image models to be creative involves using style ambiguity loss. In this work, we explore using the style ambiguity training objective, used to approximate creativity, on a diffusion model. We then experiment with forms of style ambiguity loss that do not require training a classifier or a labeled dataset, and find that the models trained with style ambiguity loss can generate better images than the baseline diffusion models and GANs. Code is available at this https URL.

Title: TPP-LLM: Modeling Temporal Point Processes by Efficiently Fine-Tuning Large Language Models

• Authors: Zefang Liu, Yinzhu Quan
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02062
• Pdf URL: https://arxiv.org/pdf/2410.02062
• Copy Paste: [[2410.02062]] TPP-LLM: Modeling Temporal Point Processes by Efficiently Fine-Tuning Large Language Models(https://arxiv.org/abs/2410.02062)
• Keywords: large language model
• Abstract: Temporal point processes (TPPs) are widely used to model the timing and occurrence of events in domains such as social networks, transportation systems, and e-commerce. In this paper, we introduce TPP-LLM, a novel framework that integrates large language models (LLMs) with TPPs to capture both the semantic and temporal aspects of event sequences. Unlike traditional methods that rely on categorical event type representations, TPP-LLM directly utilizes the textual descriptions of event types, enabling the model to capture rich semantic information embedded in the text. While LLMs excel at understanding event semantics, they are less adept at capturing temporal patterns. To address this, TPP-LLM incorporates temporal embeddings and employs parameter-efficient fine-tuning (PEFT) methods to effectively learn temporal dynamics without extensive retraining. This approach improves both predictive accuracy and computational efficiency. Experimental results across diverse real-world datasets demonstrate that TPP-LLM outperforms state-of-the-art baselines in sequence modeling and event prediction, highlighting the benefits of combining LLMs with TPPs.

Title: Inspection and Control of Self-Generated-Text Recognition Ability in Llama3-8b-Instruct

• Authors: Christopher Ackerman, Nina Panickssery
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02064
• Pdf URL: https://arxiv.org/pdf/2410.02064
• Copy Paste: [[2410.02064]] Inspection and Control of Self-Generated-Text Recognition Ability in Llama3-8b-Instruct(https://arxiv.org/abs/2410.02064)
• Keywords: robust
• Abstract: It has been reported that LLMs can recognize their own writing. As this has potential implications for AI safety, yet is relatively understudied, we investigate the phenomenon, seeking to establish whether it robustly occurs at the behavioral level, how the observed behavior is achieved, and whether it can be controlled. First, we find that the Llama3-8b-Instruct chat model - but not the base Llama3-8b model - can reliably distinguish its own outputs from those of humans, and present evidence that the chat model is likely using its experience with its own outputs, acquired during post-training, to succeed at the writing recognition task. Second, we identify a vector in the residual stream of the model that is differentially activated when the model makes a correct self-written-text recognition judgment, show that the vector activates in response to information relevant to self-authorship, present evidence that the vector is related to the concept of "self" in the model, and demonstrate that the vector is causally related to the model's ability to perceive and assert self-authorship. Finally, we show that the vector can be used to control both the model's behavior and its perception, steering the model to claim or disclaim authorship by applying the vector to the model's output as it generates it, and steering the model to believe or disbelieve it wrote arbitrary texts by applying the vector to them as the model reads them.

Title: Learning from the Giants: A Practical Approach to Underwater Depth and Surface Normals Estimation

• Authors: Alzayat Saleh, Melanie Olsen, Bouchra Senadji, Mostafa Rahimi Azghadi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02072
• Pdf URL: https://arxiv.org/pdf/2410.02072
• Copy Paste: [[2410.02072]] Learning from the Giants: A Practical Approach to Underwater Depth and Surface Normals Estimation(https://arxiv.org/abs/2410.02072)
• Keywords: transformer, generative
• Abstract: Monocular Depth and Surface Normals Estimation (MDSNE) is crucial for tasks such as 3D reconstruction, autonomous navigation, and underwater exploration. Current methods rely either on discriminative models, which struggle with transparent or reflective surfaces, or generative models, which, while accurate, are computationally expensive. This paper presents a novel deep learning model for MDSNE, specifically tailored for underwater environments, using a hybrid architecture that integrates Convolutional Neural Networks (CNNs) with Transformers, leveraging the strengths of both approaches. Training effective MDSNE models is often hampered by noisy real-world datasets and the limited generalization of synthetic datasets. To address this, we generate pseudo-labeled real data using multiple pre-trained MDSNE models. To ensure the quality of this data, we propose the Depth Normal Evaluation and Selection Algorithm (DNESA), which evaluates and selects the most reliable pseudo-labeled samples using domain-specific metrics. A lightweight student model is then trained on this curated dataset. Our model reduces parameters by 90% and training costs by 80%, allowing real-time 3D perception on resource-constrained devices. Key contributions include: a novel and efficient MDSNE model, the DNESA algorithm, a domain-specific data pipeline, and a focus on real-time performance and scalability. Designed for real-world underwater applications, our model facilitates low-cost deployments in underwater robots and autonomous vehicles, bridging the gap between research and practical implementation.

Title: Depth Pro: Sharp Monocular Metric Depth in Less Than a Second

• Authors: Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, Vladlen Koltun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02073
• Pdf URL: https://arxiv.org/pdf/2410.02073
• Copy Paste: [[2410.02073]] Depth Pro: Sharp Monocular Metric Depth in Less Than a Second(https://arxiv.org/abs/2410.02073)
• Keywords: transformer
• Abstract: We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image. Extensive experiments analyze specific design choices and demonstrate that Depth Pro outperforms prior work along multiple dimensions. We release code and weights at this https URL

Title: Kolmogorov-Arnold Network Autoencoders

• Authors: Mohammadamin Moradi, Shirin Panahi, Erik Bollt, Ying-Cheng Lai
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02077
• Pdf URL: https://arxiv.org/pdf/2410.02077
• Copy Paste: [[2410.02077]] Kolmogorov-Arnold Network Autoencoders(https://arxiv.org/abs/2410.02077)
• Keywords: interpretability
• Abstract: Deep learning models have revolutionized various domains, with Multi-Layer Perceptrons (MLPs) being a cornerstone for tasks like data regression and image classification. However, a recent study has introduced Kolmogorov-Arnold Networks (KANs) as promising alternatives to MLPs, leveraging activation functions placed on edges rather than nodes. This structural shift aligns KANs closely with the Kolmogorov-Arnold representation theorem, potentially enhancing both model accuracy and interpretability. In this study, we explore the efficacy of KANs in the context of data representation via autoencoders, comparing their performance with traditional Convolutional Neural Networks (CNNs) on the MNIST, SVHN, and CIFAR-10 datasets. Our results demonstrate that KAN-based autoencoders achieve competitive performance in terms of reconstruction accuracy, thereby suggesting their viability as effective tools in data analysis tasks.

Title: Deep Generative Modeling for Identification of Noisy, Non-Stationary Dynamical Systems

• Authors: Doris Voina, Steven Brunton, J. Nathan Kutz
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.02079
• Pdf URL: https://arxiv.org/pdf/2410.02079
• Copy Paste: [[2410.02079]] Deep Generative Modeling for Identification of Noisy, Non-Stationary Dynamical Systems(https://arxiv.org/abs/2410.02079)
• Keywords: generative
• Abstract: A significant challenge in many fields of science and engineering is making sense of time-dependent measurement data by recovering governing equations in the form of differential equations. We focus on finding parsimonious ordinary differential equation (ODE) models for nonlinear, noisy, and non-autonomous dynamical systems and propose a machine learning method for data-driven system identification. While many methods tackle noisy and limited data, non-stationarity - where differential equation parameters change over time - has received less attention. Our method, dynamic SINDy, combines variational inference with SINDy (sparse identification of nonlinear dynamics) to model time-varying coefficients of sparse ODEs. This framework allows for uncertainty quantification of ODE coefficients, expanding on previous methods for autonomous systems. These coefficients are then interpreted as latent variables and added to the system to obtain an autonomous dynamical model. We validate our approach using synthetic data, including nonlinear oscillators and the Lorenz system, and apply it to neuronal activity data from C. elegans. Dynamic SINDy uncovers a global nonlinear model, showing it can handle real, noisy, and chaotic datasets. We aim to apply our method to a variety of problems, specifically dynamic systems with complex time-dependent parameters.

Title: EMMA: Efficient Visual Alignment in Multi-Modal LLMs

• Authors: Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Siddharth Garg, Farshad Khorrami
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02080
• Pdf URL: https://arxiv.org/pdf/2410.02080
• Copy Paste: [[2410.02080]] EMMA: Efficient Visual Alignment in Multi-Modal LLMs(https://arxiv.org/abs/2410.02080)
• Keywords: robust, interpretability, large language model
• Abstract: Multi-modal Large Language Models (MLLMs) have recently exhibited impressive general-purpose capabilities by leveraging vision foundation models to encode the core concepts of images into representations. These are then combined with instructions and processed by the language model to generate high-quality responses. Despite significant progress in enhancing the language component, challenges persist in optimally fusing visual encodings within the language model for task-specific adaptability. Recent research has focused on improving this fusion through modality adaptation modules but at the cost of significantly increased model complexity and training data needs. In this paper, we propose EMMA (Efficient Multi-Modal Adaptation), a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model. Our key contributions include: (1) an efficient early fusion mechanism that integrates vision and language representations with minimal added parameters (less than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light on the internal mechanisms of the proposed method; (3) comprehensive experiments that demonstrate notable improvements on both specialized and general benchmarks for MLLMs. Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations. Our code is available at this https URL

Title: MixLinear: Extreme Low Resource Multivariate Time Series Forecasting with 0.1K Parameters

• Authors: Aitian Ma, Dongsheng Luo, Mo Sha
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02081
• Pdf URL: https://arxiv.org/pdf/2410.02081
• Copy Paste: [[2410.02081]] MixLinear: Extreme Low Resource Multivariate Time Series Forecasting with 0.1K Parameters(https://arxiv.org/abs/2410.02081)
• Keywords: transformer
• Abstract: Recently, there has been a growing interest in Long-term Time Series Forecasting (LTSF), which involves predicting long-term future values by analyzing a large amount of historical time-series data to identify patterns and trends. There exist significant challenges in LTSF due to its complex temporal dependencies and high computational demands. Although Transformer-based models offer high forecasting accuracy, they are often too compute-intensive to be deployed on devices with hardware constraints. On the other hand, the linear models aim to reduce the computational overhead by employing either decomposition methods in the time domain or compact representations in the frequency domain. In this paper, we propose MixLinear, an ultra-lightweight multivariate time series forecasting model specifically designed for resource-constrained devices. MixLinear effectively captures both temporal and frequency domain features by modeling intra-segment and inter-segment variations in the time domain and extracting frequency variations from a low-dimensional latent space in the frequency domain. By reducing the parameter scale of a downsampled $n$-length input/output one-layer linear model from $O(n^2)$ to $O(n)$, MixLinear achieves efficient computation without sacrificing accuracy. Extensive evaluations with four benchmark datasets show that MixLinear attains forecasting performance comparable to, or surpassing, state-of-the-art models with significantly fewer parameters ($0.1K$), which makes it well-suited for deployment on devices with limited computational capacity.

Title: Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations

• Authors: Minoh Jeong, Min Namgung, Zae Myung Kim, Dongyeop Kang, Yao-Yi Chiang, Alfred Hero
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02086
• Pdf URL: https://arxiv.org/pdf/2410.02086
• Copy Paste: [[2410.02086]] Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations(https://arxiv.org/abs/2410.02086)
• Keywords: robust
• Abstract: Multimodal learning plays a crucial role in enabling machine learning models to fuse and utilize diverse data sources, such as text, images, and audio, to support a variety of downstream tasks. A unified representation across various modalities is particularly important for improving efficiency and performance. Recent binding methods, such as ImageBind (Girdhar et al., 2023), typically use a fixed anchor modality to align multimodal data in the anchor modal embedding space. In this paper, we mathematically analyze the fixed anchor binding methods and uncover notable limitations: (1) over-reliance on the choice of the anchor modality, (2) failure to capture intra-modal information, and (3) failure to account for inter-modal correlation among non-anchored modalities. To address these limitations, we propose CentroBind, a simple yet powerful approach that eliminates the need for a fixed anchor; instead, it employs dynamically adjustable centroid-based anchors generated from all available modalities, resulting in a balanced and rich representation space. We theoretically demonstrate that our method captures three crucial properties of multimodal learning: intra-modal learning, inter-modal learning, and multimodal alignment, while also constructing a robust unified representation across all modalities. Our experiments on both synthetic and real-world datasets demonstrate the superiority of the proposed method, showing that dynamic anchor methods outperform all fixed anchor binding methods as the former captures more nuanced multimodal interactions.

Title: RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning

• Authors: Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, Gabriel Synnaeve
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02089
• Pdf URL: https://arxiv.org/pdf/2410.02089
• Copy Paste: [[2410.02089]] RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning(https://arxiv.org/abs/2410.02089)
• Keywords: large language model
• Abstract: Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engagement to a minimum. Crucially, such LLMs need to ground their generations in any feedback obtained to reliably achieve desired outcomes. We propose an end-to-end reinforcement learning method for teaching models to leverage execution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sampling. We benchmark on competitive programming tasks, where we achieve new start-of-the art results with both small (8B parameters) and large (70B) models while reducing the amount of samples required by an order of magnitude. Our analysis of inference-time behavior demonstrates that our method produces LLMs that effectively leverage automatic feedback over multiple steps.

Title: DomainLynx: Leveraging Large Language Models for Enhanced Domain Squatting Detection

• Authors: Daiki Chiba, Hiroki Nakano, Takashi Koide
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02095
• Pdf URL: https://arxiv.org/pdf/2410.02095
• Copy Paste: [[2410.02095]] DomainLynx: Leveraging Large Language Models for Enhanced Domain Squatting Detection(https://arxiv.org/abs/2410.02095)
• Keywords: security, protect, attack, robust, large language model
• Abstract: Domain squatting poses a significant threat to Internet security, with attackers employing increasingly sophisticated techniques. This study introduces DomainLynx, an innovative compound AI system leveraging Large Language Models (LLMs) for enhanced domain squatting detection. Unlike existing methods focusing on predefined patterns for top-ranked domains, DomainLynx excels in identifying novel squatting techniques and protecting less prominent brands. The system's architecture integrates advanced data processing, intelligent domain pairing, and LLM-powered threat assessment. Crucially, DomainLynx incorporates specialized components that mitigate LLM hallucinations, ensuring reliable and context-aware detection. This approach enables efficient analysis of vast security data from diverse sources, including Certificate Transparency logs, Passive DNS records, and zone files. Evaluated on a curated dataset of 1,649 squatting domains, DomainLynx achieved 94.7\% accuracy using Llama-3-70B. In a month-long real-world test, it detected 34,359 squatting domains from 2.09 million new domains, outperforming baseline methods by 2.5 times. This research advances Internet security by providing a versatile, accurate, and adaptable tool for combating evolving domain squatting threats. DomainLynx's approach paves the way for more robust, AI-driven cybersecurity solutions, enhancing protection for a broader range of online entities and contributing to a safer digital ecosystem.

Title: DomainDynamics: Lifecycle-Aware Risk Timeline Construction for Domain Names

• Authors: Daiki Chiba, Hiroki Nakano, Takashi Koide
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02096
• Pdf URL: https://arxiv.org/pdf/2410.02096
• Copy Paste: [[2410.02096]] DomainDynamics: Lifecycle-Aware Risk Timeline Construction for Domain Names(https://arxiv.org/abs/2410.02096)
• Keywords: attack
• Abstract: The persistent threat posed by malicious domain names in cyber-attacks underscores the urgent need for effective detection mechanisms. Traditional machine learning methods, while capable of identifying such domains, often suffer from high false positive and false negative rates due to their extensive reliance on historical data. Conventional approaches often overlook the dynamic nature of domain names, the purposes and ownership of which may evolve, potentially rendering risk assessments outdated or irrelevant. To address these shortcomings, we introduce DomainDynamics, a novel system designed to predict domain name risks by considering their lifecycle stages. DomainDynamics constructs a timeline for each domain, evaluating the characteristics of each domain at various points in time to make informed, temporal risk determinations. In an evaluation experiment involving over 85,000 actual malicious domains from malware and phishing incidents, DomainDynamics demonstrated a significant improvement in detection rates, achieving an 82.58\% detection rate with a low false positive rate of 0.41\%. This performance surpasses that of previous studies and commercial services, improving detection capability substantially.

Title: DomainHarvester: Harvesting Infrequently Visited Yet Trustworthy Domain Names

• Authors: Daiki Chiba, Hiroki Nakano, Takashi Koide
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02097
• Pdf URL: https://arxiv.org/pdf/2410.02097
• Copy Paste: [[2410.02097]] DomainHarvester: Harvesting Infrequently Visited Yet Trustworthy Domain Names(https://arxiv.org/abs/2410.02097)
• Keywords: security, transformer
• Abstract: In cybersecurity, allow lists play a crucial role in distinguishing safe websites from potential threats. Conventional methods for compiling allow lists, focusing heavily on website popularity, often overlook infrequently visited legitimate domains. This paper introduces DomainHarvester, a system aimed at generating allow lists that include trustworthy yet infrequently visited domains. By adopting an innovative bottom-up methodology that leverages the web's hyperlink structure, DomainHarvester identifies legitimate yet underrepresented domains. The system uses seed URLs to gather domain names, employing machine learning with a Transformer-based approach to assess their trustworthiness. DomainHarvester has developed two distinct allow lists: one with a global focus and another emphasizing local relevance. Compared to six existing top lists, DomainHarvester's allow lists show minimal overlaps, 4\% globally and 0.1\% locally, while significantly reducing the risk of including malicious domains, thereby enhancing security. The contributions of this research are substantial, illuminating the overlooked aspect of trustworthy yet underrepresented domains and introducing DomainHarvester, a system that goes beyond traditional popularity-based metrics. Our methodology enhances the inclusivity and precision of allow lists, offering significant advantages to users and businesses worldwide, especially in non-English speaking regions.

Title: EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing

• Authors: Haotian Sun, Bowen Zhang, Yanghao Li, Haoshuo Huang, Tao Lei, Ruoming Pang, Bo Dai, Nan Du
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02098
• Pdf URL: https://arxiv.org/pdf/2410.02098
• Copy Paste: [[2410.02098]] EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing(https://arxiv.org/abs/2410.02098)
• Keywords: interpretability, diffusion, transformer
• Abstract: Diffusion transformers have been widely adopted for text-to-image synthesis. While scaling these models up to billions of parameters shows promise, the effectiveness of scaling beyond current sizes remains underexplored and challenging. By explicitly exploiting the computational heterogeneity of image generations, we develop a new family of Mixture-of-Experts (MoE) models (EC-DIT) for diffusion transformers with expert-choice routing. EC-DIT learns to adaptively optimize the compute allocated to understand the input texts and generate the respective image patches, enabling heterogeneous computation aligned with varying text-image complexities. This heterogeneity provides an efficient way of scaling EC-DIT up to 97 billion parameters and achieving significant improvements in training convergence, text-to-image alignment, and overall generation quality over dense models and conventional MoE models. Through extensive ablations, we show that EC-DIT demonstrates superior scalability and adaptive compute allocation by recognizing varying textual importance through end-to-end training. Notably, in text-to-image alignment evaluation, our largest models achieve a state-of-the-art GenEval score of 71.68% and still maintain competitive inference speed with intuitive interpretability.

Title: A Watermark for Black-Box Language Models

• Authors: Dara Bahri, John Wieting, Dana Alon, Donald Metzler
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02099
• Pdf URL: https://arxiv.org/pdf/2410.02099
• Copy Paste: [[2410.02099]] A Watermark for Black-Box Language Models(https://arxiv.org/abs/2410.02099)
• Keywords: watermark, large language model
• Abstract: Watermarking has recently emerged as an effective strategy for detecting the outputs of large language models (LLMs). Most existing schemes require \emph{white-box} access to the model's next-token probability distribution, which is typically not accessible to downstream users of an LLM API. In this work, we propose a principled watermarking scheme that requires only the ability to sample sequences from the LLM (i.e. \emph{black-box} access), boasts a \emph{distortion-free} property, and can be chained or nested using multiple secret keys. We provide performance guarantees, demonstrate how it can be leveraged when white-box access is available, and show when it can outperform existing white-box schemes via comprehensive experiments.

Title: Racing Thoughts: Explaining Large Language Model Contextualization Errors

• Authors: Michael A. Lepori, Michael Mozer, Asma Ghandeharioun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02102
• Pdf URL: https://arxiv.org/pdf/2410.02102
• Copy Paste: [[2410.02102]] Racing Thoughts: Explaining Large Language Model Contextualization Errors(https://arxiv.org/abs/2410.02102)
• Keywords: transformer, large language model
• Abstract: The profound success of transformer-based language models can largely be attributed to their ability to integrate relevant contextual information from an input sequence in order to generate a response or complete a task. However, we know very little about the algorithms that a model employs to implement this capability, nor do we understand their failure modes. For example, given the prompt "John is going fishing, so he walks over to the bank. Can he make an ATM transaction?", a model may incorrectly respond "Yes" if it has not properly contextualized "bank" as a geographical feature, rather than a financial institution. We propose the LLM Race Conditions Hypothesis as an explanation of contextualization errors of this form. This hypothesis identifies dependencies between tokens (e.g., "bank" must be properly contextualized before the final token, "?", integrates information from "bank"), and claims that contextualization errors are a result of violating these dependencies. Using a variety of techniques from mechanistic intepretability, we provide correlational and causal evidence in support of the hypothesis, and suggest inference-time interventions to address it.

Title: ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement

• Authors: Xiangyu Peng, Congying Xia, Xinyi Yang, Caiming Xiong, Chien-Sheng Wu, Chen Xing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02108
• Pdf URL: https://arxiv.org/pdf/2410.02108
• Copy Paste: [[2410.02108]] ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement(https://arxiv.org/abs/2410.02108)
• Keywords: large language model
• Abstract: Post-training Large Language Models (LLMs) with explicit reasoning trajectories can enhance their reasoning abilities. However, acquiring such high-quality trajectory data typically demands meticulous supervision from humans or superior models, which can be either expensive or license-constrained. In this paper, we explore how far an LLM can improve its reasoning by self-synthesizing reasoning paths as training data without any additional supervision. Existing self-synthesizing methods, such as STaR, suffer from poor generalization to out-of-domain (OOD) reasoning tasks. We hypothesize it is due to that their self-synthesized reasoning paths are too task-specific, lacking general task-agnostic reasoning guidance. To address this, we propose Reasoning Generalist via Self-Improvement (ReGenesis), a method to self-synthesize reasoning paths as post-training data by progressing from abstract to concrete. More specifically, ReGenesis self-synthesizes reasoning paths by converting general reasoning guidelines into task-specific ones, generating reasoning structures, and subsequently transforming these structures into reasoning paths, without the need for human-designed task-specific examples used in existing methods. We show that ReGenesis achieves superior performance on all in-domain and OOD settings tested compared to existing methods. For six OOD tasks specifically, while previous methods exhibited an average performance decrease of approximately 4.6% after post training, ReGenesis delivers around 6.1% performance improvement. We also conduct in-depth analysis of our framework and show ReGenesis is effective across various LLMs and design choices.

Title: Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEs

• Authors: Chun-Wun Cheng, Jiahao Huang, Yi Zhang, Guang Yang, Carola-Bibiane Schönlieb, Angelica I Aviles-Rivero
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2410.02113
• Pdf URL: https://arxiv.org/pdf/2410.02113
• Copy Paste: [[2410.02113]] Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEs(https://arxiv.org/abs/2410.02113)
• Keywords: transformer
• Abstract: Partial differential equations (PDEs) are widely used to model complex physical systems, but solving them efficiently remains a significant challenge. Recently, Transformers have emerged as the preferred architecture for PDEs due to their ability to capture intricate dependencies. However, they struggle with representing continuous dynamics and long-range interactions. To overcome these limitations, we introduce the Mamba Neural Operator (MNO), a novel framework that enhances neural operator-based techniques for solving PDEs. MNO establishes a formal theoretical connection between structured state-space models (SSMs) and neural operators, offering a unified structure that can adapt to diverse architectures, including Transformer-based models. By leveraging the structured design of SSMs, MNO captures long-range dependencies and continuous dynamics more effectively than traditional Transformers. Through extensive analysis, we show that MNO significantly boosts the expressive power and accuracy of neural operators, making it not just a complement but a superior framework for PDE-related tasks, bridging the gap between efficient representation and accurate solution approximation.

Title: C-MELT: Contrastive Enhanced Masked Auto-Encoders for ECG-Language Pre-Training

• Authors: Manh Pham, Aaqib Saeed, Dong Ma
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02131
• Pdf URL: https://arxiv.org/pdf/2410.02131
• Copy Paste: [[2410.02131]] C-MELT: Contrastive Enhanced Masked Auto-Encoders for ECG-Language Pre-Training(https://arxiv.org/abs/2410.02131)
• Keywords: robust, generative
• Abstract: Accurate interpretation of Electrocardiogram (ECG) signals is pivotal for diagnosing cardiovascular diseases. Integrating ECG signals with their accompanying textual reports holds immense potential to enhance clinical diagnostics through the combination of physiological data and qualitative insights. However, this integration faces significant challenges due to inherent modality disparities and the scarcity of labeled data for robust cross-modal learning. To address these obstacles, we propose C-MELT, a novel framework that pre-trains ECG and text data using a contrastive masked auto-encoder architecture. C-MELT uniquely combines the strengths of generative with enhanced discriminative capabilities to achieve robust cross-modal representations. This is accomplished through masked modality modeling, specialized loss functions, and an improved negative sampling strategy tailored for cross-modal alignment. Extensive experiments on five public datasets across diverse downstream tasks demonstrate that C-MELT significantly outperforms existing methods, achieving 15% and 2% increases in linear probing and zero-shot performance over state-of-the-art models, respectively. These results highlight the effectiveness of C-MELT, underscoring its potential to advance automated clinical diagnostics through multi-modal representations.

Title: TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysis

• Authors: Ziyang Song, Qingcheng Lu, He Zhu, David Buckeridge, Yue Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02133
• Pdf URL: https://arxiv.org/pdf/2410.02133
• Copy Paste: [[2410.02133]] TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysis(https://arxiv.org/abs/2410.02133)
• Keywords: transformer, generative
• Abstract: In many domains, such as healthcare, time-series data is often irregularly sampled with varying intervals between observations. This poses challenges for classical time-series models that require equally spaced data. To address this, we propose a novel time-series Transformer called Trajectory Generative Pre-trained Transformer (TrajGPT). TrajGPT employs a novel Selective Recurrent Attention (SRA) mechanism, which utilizes a data-dependent decay to adaptively filter out irrelevant past information based on contexts. By interpreting TrajGPT as discretized ordinary differential equations (ODEs), it effectively captures the underlying continuous dynamics and enables time-specific inference for forecasting arbitrary target timesteps. Experimental results demonstrate that TrajGPT excels in trajectory forecasting, drug usage prediction, and phenotype classification without requiring task-specific fine-tuning. By evolving the learned continuous dynamics, TrajGPT can interpolate and extrapolate disease risk trajectories from partially-observed time series. The visualization of predicted health trajectories shows that TrajGPT forecasts unseen diseases based on the history of clinically relevant phenotypes (i.e., contexts).

Title: Disentangled Representation Learning for Parametric Partial Differential Equations

• Authors: Ning Liu, Lu Zhang, Tian Gao, Yue Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02136
• Pdf URL: https://arxiv.org/pdf/2410.02136
• Copy Paste: [[2410.02136]] Disentangled Representation Learning for Parametric Partial Differential Equations(https://arxiv.org/abs/2410.02136)
• Keywords: robust, interpretability
• Abstract: Neural operators (NOs) have demonstrated remarkable success in learning mappings between function spaces, serving as efficient approximators for the forward solutions of complex physical systems governed by partial differential equations (PDEs). However, while effective as black-box solvers, they offer limited insight into the underlying physical mechanism, due to the lack of interpretable representations of the physical parameters that drive the system. To tackle this challenge, we propose a new paradigm for learning disentangled representations from neural operator parameters, thereby effectively solving an inverse problem. Specifically, we introduce DisentangO, a novel hyper-neural operator architecture designed to unveil and disentangle the latent physical factors of variation embedded within the black-box neural operator parameters. At the core of DisentangO is a multi-task neural operator architecture that distills the varying parameters of the governing PDE through a task-wise adaptive layer, coupled with a hierarchical variational autoencoder that disentangles these variations into identifiable latent factors. By learning these disentangled representations, our model not only enhances physical interpretability but also enables more robust generalization across diverse physical systems. Empirical evaluations across supervised, semi-supervised, and unsupervised learning contexts show that DisentangO effectively extracts meaningful and interpretable latent features, bridging the divide between predictive performance and physical understanding in neural operator frameworks.

Title: A Formal Framework for Understanding Length Generalization in Transformers

• Authors: Xinting Huang, Andy Yang, Satwik Bhattamishra, Yash Sarrof, Andreas Krebs, Hattie Zhou, Preetum Nakkiran, Michael Hahn
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02140
• Pdf URL: https://arxiv.org/pdf/2410.02140
• Copy Paste: [[2410.02140]] A Formal Framework for Understanding Length Generalization in Transformers(https://arxiv.org/abs/2410.02140)
• Keywords: transformer
• Abstract: A major challenge for transformers is generalizing to sequences longer than those observed during training. While previous works have empirically shown that transformers can either succeed or fail at length generalization depending on the task, theoretical understanding of this phenomenon remains limited. In this work, we introduce a rigorous theoretical framework to analyze length generalization in causal transformers with learnable absolute positional encodings. In particular, we characterize those functions that are identifiable in the limit from sufficiently long inputs with absolute positional encodings under an idealized inference scheme using a norm-based regularizer. This enables us to prove the possibility of length generalization for a rich family of problems. We experimentally validate the theory as a predictor of success and failure of length generalization across a range of algorithmic and formal language tasks. Our theory not only explains a broad set of empirical observations but also opens the way to provably predicting length generalization capabilities in transformers.

Title: Plug-and-Play Controllable Generation for Discrete Masked Models

• Authors: Wei Guo, Yuchen Zhu, Molei Tao, Yongxin Chen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02143
• Pdf URL: https://arxiv.org/pdf/2410.02143
• Copy Paste: [[2410.02143]] Plug-and-Play Controllable Generation for Discrete Masked Models(https://arxiv.org/abs/2410.02143)
• Keywords: generative
• Abstract: This article makes discrete masked models for the generative modeling of discrete data controllable. The goal is to generate samples of a discrete random variable that adheres to a posterior distribution, satisfies specific constraints, or optimizes a reward function. This methodological development enables broad applications across downstream tasks such as class-specific image generation and protein design. Existing approaches for controllable generation of masked models typically rely on task-specific fine-tuning or additional modifications, which can be inefficient and resource-intensive. To overcome these limitations, we propose a novel plug-and-play framework based on importance sampling that bypasses the need for training a conditional score. Our framework is agnostic to the choice of control criteria, requires no gradient information, and is well-suited for tasks such as posterior sampling, Bayesian inverse problems, and constrained generation. We demonstrate the effectiveness of our approach through extensive experiments, showcasing its versatility across multiple domains, including protein design.

Title: An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos

• Authors: Arun Reddy, Ketul Shah, Corban Rivera, William Paul, Celso M. De Melo, Rama Chellappa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02152
• Pdf URL: https://arxiv.org/pdf/2410.02152
• Copy Paste: [[2410.02152]] An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos(https://arxiv.org/abs/2410.02152)
• Keywords: robust
• Abstract: In this work, we explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We consider whether these models have sufficiently robust and expressive representation spaces to enable "training-free" classification. Specifically, we utilize various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only. We compare these results with another training-free approach -- zero-shot classification using text descriptions of each gesture. In our experiments with the RoCoG-v2 dataset, we find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos. We also observe that video backbones that were fine-tuned on classification tasks serve as superior feature extractors, and that the choice of fine-tuning data has a substantial impact on k-nearest neighbors performance. Lastly, we find that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language.

Title: ClassContrast: Bridging the Spatial and Contextual Gaps for Node Representations

• Authors: Md Joshem Uddin, Astrit Tola, Varin Sikand, Cuneyt Gurcan Akcora, Baris Coskunuzer
• Subjects: cs.LG, cs.CG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02158
• Pdf URL: https://arxiv.org/pdf/2410.02158
• Copy Paste: [[2410.02158]] ClassContrast: Bridging the Spatial and Contextual Gaps for Node Representations(https://arxiv.org/abs/2410.02158)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have revolutionized the domain of graph representation learning by utilizing neighborhood aggregation schemes in many popular architectures, such as message passing graph neural networks (MPGNNs). This scheme involves iteratively calculating a node's representation vector by aggregating and transforming the representation vectors of its adjacent nodes. Despite their effectiveness, MPGNNs face significant issues, such as oversquashing, oversmoothing, and underreaching, which hamper their effectiveness. Additionally, the reliance of MPGNNs on the homophily assumption, where edges typically connect nodes with similar labels and features, limits their performance in heterophilic contexts, where connected nodes often have significant differences. This necessitates the development of models that can operate effectively in both homophilic and heterophilic settings. In this paper, we propose a novel approach, ClassContrast, grounded in Energy Landscape Theory from Chemical Physics, to overcome these limitations. ClassContrast combines spatial and contextual information, leveraging a physics-inspired energy landscape to model node embeddings that are both discriminative and robust across homophilic and heterophilic settings. Our approach introduces contrast-based homophily matrices to enhance the understanding of class interactions and tendencies. Through extensive experiments, we demonstrate that ClassContrast outperforms traditional GNNs in node classification and link prediction tasks, proving its effectiveness and versatility in diverse real-world scenarios.

Title: Controlled Generation of Natural Adversarial Documents for Stealthy Retrieval Poisoning

• Authors: Collin Zhang, Tingwei Zhang, Vitaly Shmatikov
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02163
• Pdf URL: https://arxiv.org/pdf/2410.02163
• Copy Paste: [[2410.02163]] Controlled Generation of Natural Adversarial Documents for Stealthy Retrieval Poisoning(https://arxiv.org/abs/2410.02163)
• Keywords: steal
• Abstract: Recent work showed that retrieval based on embedding similarity (e.g., for retrieval-augmented generation) is vulnerable to poisoning: an adversary can craft malicious documents that are retrieved in response to broad classes of queries. We demonstrate that previous, HotFlip-based techniques produce documents that are very easy to detect using perplexity filtering. Even if generation is constrained to produce low-perplexity text, the resulting documents are recognized as unnatural by LLMs and can be automatically filtered from the retrieval corpus. We design, implement, and evaluate a new controlled generation technique that combines an adversarial objective (embedding similarity) with a "naturalness" objective based on soft scores computed using an open-source, surrogate LLM. The resulting adversarial documents (1) cannot be automatically detected using perplexity filtering and/or other LLMs, except at the cost of significant false positives in the retrieval corpus, yet (2) achieve similar poisoning efficacy to easily-detectable documents generated using HotFlip, and (3) are significantly more effective than prior methods for energy-guided generation, such as COLD.

Title: Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis

• Authors: Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, Pin-Yu Chen
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02167
• Pdf URL: https://arxiv.org/pdf/2410.02167
• Copy Paste: [[2410.02167]] Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis(https://arxiv.org/abs/2410.02167)
• Keywords: transformer, large language model
• Abstract: Chain-of-Thought (CoT) is an efficient prompting method that enables the reasoning ability of large language models by augmenting the query using multiple examples with multiple intermediate steps. Despite the empirical success, the theoretical understanding of how to train a Transformer to achieve the CoT ability remains less explored. This is primarily due to the technical challenges involved in analyzing the nonconvex optimization on nonlinear attention models. To the best of our knowledge, this work provides the first theoretical study of training Transformers with nonlinear attention to obtain the CoT generalization capability so that the resulting model can inference on unseen tasks when the input is augmented by examples of the new task. We first quantify the required training samples and iterations to train a Transformer model towards CoT ability. We then prove the success of its CoT generalization on unseen tasks with distribution-shifted testing data. Moreover, we theoretically characterize the conditions for an accurate reasoning output by CoT even when the provided reasoning examples contain noises and are not always accurate. In contrast, in-context learning (ICL), which can be viewed as one-step CoT without intermediate steps, may fail to provide an accurate output when CoT does. These theoretical findings are justified through experiments.

Title: Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecasting

• Authors: Siyang Li, Yize Chen, Hui Xiong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02168
• Pdf URL: https://arxiv.org/pdf/2410.02168
• Copy Paste: [[2410.02168]] Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecasting(https://arxiv.org/abs/2410.02168)
• Keywords: diffusion, generative
• Abstract: Forecasting faithful trajectories of multivariate time series from practical scopes is essential for reasonable decision-making. Recent methods majorly tailor generative conditional diffusion models to estimate the target temporal predictive distribution. However, it remains an obstacle to enhance the exploitation efficiency of given implicit temporal predictive information to bolster conditional diffusion learning. To this end, we propose a generic channel-aware Contrastive Conditional Diffusion model entitled CCDM to achieve desirable Multivariate probabilistic forecasting, obviating the need for curated temporal conditioning inductive biases. In detail, we first design a channel-centric conditional denoising network to manage intra-variate variations and cross-variate correlations, which can lead to scalability on diverse prediction horizons and channel numbers. Then, we devise an ad-hoc denoising-based temporal contrastive learning to explicitly amplify the predictive mutual information between past observations and future forecasts. It can coherently complement naive step-wise denoising diffusion training and improve the forecasting accuracy and generality on unknown test time series. Besides, we offer theoretic insights on the benefits of such auxiliary contrastive training refinement from both neural mutual information and temporal distribution generalization aspects. The proposed CCDM can exhibit superior forecasting capability compared to current state-of-the-art diffusion forecasters over a comprehensive benchmark, with best MSE and CRPS outcomes on $66.67\%$ and $83.33\%$ cases. Our code is publicly available at this https URL.

Title: Efficiently Deploying LLMs with Controlled Risk

• Authors: Michael J. Zellinger, Matt Thomson
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02173
• Pdf URL: https://arxiv.org/pdf/2410.02173
• Copy Paste: [[2410.02173]] Efficiently Deploying LLMs with Controlled Risk(https://arxiv.org/abs/2410.02173)
• Keywords: large language model
• Abstract: Deploying large language models in production requires simultaneous attention to efficiency and risk control. Prior work has shown the possibility to cut costs while maintaining similar accuracy, but has neglected to focus on risk control. By contrast, here we present hierarchical chains with multi-level abstention (HCMA), which use model-intrinsic uncertainty to delegate queries along the LLM intelligence hierarchy, enabling training-free model switching based solely on black-box API calls. Our framework presents novel trade-offs between efficiency and risk. For example, deploying HCMA on MMLU cuts the error rate of Llama3 405B by 30% when the model is allowed to abstain on 20% of the queries. To calibrate HCMA for optimal performance, our approach uses data-efficient logistic regressions (based on a simple nonlinear feature transformation), which require only 50 or 100 labeled examples to achieve excellent calibration error (ECE), cutting ECE by 50% compared to naive Platt scaling. On free-form generation tasks, we find that chain-of-thought is ineffectual for selective prediction, whereas zero-shot prompting drives error to 0% on TruthfulQA at high abstention rates. As LLMs are increasingly deployed across computing environments with different capabilities (such as mobile, laptop, and cloud), our framework paves the way towards maintaining deployment efficiency while putting in place sharp risk controls.

Title: HATFormer: Historic Handwritten Arabic Text Recognition with Transformers

• Authors: Adrian Chan, Anupam Mijar, Mehreen Saeed, Chau-Wai Wong, Akram Khater
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02179
• Pdf URL: https://arxiv.org/pdf/2410.02179
• Copy Paste: [[2410.02179]] HATFormer: Historic Handwritten Arabic Text Recognition with Transformers(https://arxiv.org/abs/2410.02179)
• Keywords: transformer
• Abstract: Arabic handwritten text recognition (HTR) is challenging, especially for historical texts, due to diverse writing styles and the intrinsic features of Arabic script. Additionally, Arabic handwriting datasets are smaller compared to English ones, making it difficult to train generalizable Arabic HTR models. To address these challenges, we propose HATFormer, a transformer-based encoder-decoder architecture that builds on a state-of-the-art English HTR model. By leveraging the transformer's attention mechanism, HATFormer captures spatial contextual information to address the intrinsic challenges of Arabic script through differentiating cursive characters, decomposing visual representations, and identifying diacritics. Our customization to historical handwritten Arabic includes an image processor for effective ViT information preprocessing, a text tokenizer for compact Arabic text representation, and a training pipeline that accounts for a limited amount of historic Arabic handwriting data. HATFormer achieves a character error rate (CER) of 8.6% on the largest public historical handwritten Arabic dataset, with a 51% improvement over the best baseline in the literature. HATFormer also attains a comparable CER of 4.2% on the largest private non-historical dataset. Our work demonstrates the feasibility of adapting an English HTR method to a low-resource language with complex, language-specific challenges, contributing to advancements in document digitization, information retrieval, and cultural preservation.

Title: BadCM: Invisible Backdoor Attack Against Cross-Modal Learning

• Authors: Zheng Zhang, Xu Yuan, Lei Zhu, Jingkuan Song, Liqiang Nie
• Subjects: cs.CV, cs.CR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.02182
• Pdf URL: https://arxiv.org/pdf/2410.02182
• Copy Paste: [[2410.02182]] BadCM: Invisible Backdoor Attack Against Cross-Modal Learning(https://arxiv.org/abs/2410.02182)
• Keywords: defense, attack, robust, steal
• Abstract: Despite remarkable successes in unimodal learning tasks, backdoor attacks against cross-modal learning are still underexplored due to the limited generalization and inferior stealthiness when involving multiple modalities. Notably, since works in this area mainly inherit ideas from unimodal visual attacks, they struggle with dealing with diverse cross-modal attack circumstances and manipulating imperceptible trigger samples, which hinders their practicability in real-world applications. In this paper, we introduce a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor and propose a generalized invisible backdoor framework against cross-modal learning (BadCM). Specifically, a cross-modal mining scheme is developed to capture the modality-invariant components as target poisoning areas, where well-designed trigger patterns injected into these regions can be efficiently recognized by the victim models. This strategy is adapted to different image-text cross-modal models, making our framework available to various attack scenarios. Furthermore, for generating poisoned samples of high stealthiness, we conceive modality-specific generators for visual and linguistic modalities that facilitate hiding explicit trigger patterns in modality-invariant regions. To the best of our knowledge, BadCM is the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework. Comprehensive experimental evaluations on two typical applications, i.e., cross-modal retrieval and VQA, demonstrate the effectiveness and generalization of our method under multiple kinds of attack scenarios. Moreover, we show that BadCM can robustly evade existing backdoor defenses. Our code is available at this https URL.

Title: CodeJudge: Evaluating Code Generation with Large Language Models

• Authors: Weixi Tong, Tianyi Zhang
• Subjects: cs.LG, cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.02184
• Pdf URL: https://arxiv.org/pdf/2410.02184
• Copy Paste: [[2410.02184]] CodeJudge: Evaluating Code Generation with Large Language Models(https://arxiv.org/abs/2410.02184)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown promising performance in code generation. However, how to reliably evaluate code generated by LLMs remains an unresolved problem. This paper presents CodeJudge, a code evaluation framework that leverages LLMs to evaluate the semantic correctness of generated code without the need for test cases. We investigate different ways to guide the LLM in performing "slow thinking" to arrive at an in-depth and reliable evaluation. We experimented with four LLMs as evaluators on four code generation datasets and five programming languages. The results show that CodeJudge significantly outperformed existing methods in most settings. Furthermore, compared with a SOTA GPT-3.5-based code evaluation method, CodeJudge achieved better results even when using a much smaller model, Llama-3-8B-Instruct. Our code and datasets are available on GitHub this https URL.

Title: POSIX: A Prompt Sensitivity Index For Large Language Models

• Authors: Anwoy Chatterjee, H S V N S Kowndinya Renduchintala, Sumit Bhatia, Tanmoy Chakraborty
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02185
• Pdf URL: https://arxiv.org/pdf/2410.02185
• Copy Paste: [[2410.02185]] POSIX: A Prompt Sensitivity Index For Large Language Models(https://arxiv.org/abs/2410.02185)
• Keywords: large language model
• Abstract: Despite their remarkable capabilities, Large Language Models (LLMs) are found to be surprisingly sensitive to minor variations in prompts, often generating significantly divergent outputs in response to minor variations in the prompts, such as spelling errors, alteration of wording or the prompt template. However, while assessing the quality of an LLM, the focus often tends to be solely on its performance on downstream tasks, while very little to no attention is paid to prompt sensitivity. To fill this gap, we propose POSIX - a novel PrOmpt Sensitivity IndeX as a reliable measure of prompt sensitivity, thereby offering a more comprehensive evaluation of LLM performance. The key idea behind POSIX is to capture the relative change in loglikelihood of a given response upon replacing the corresponding prompt with a different intent-preserving prompt. We provide thorough empirical evidence demonstrating the efficacy of POSIX in capturing prompt sensitivity and subsequently use it to measure and thereby compare prompt sensitivity of various open-source LLMs. We find that merely increasing the parameter count or instruction tuning does not necessarily reduce prompt sensitivity whereas adding some few-shot exemplars, even just one, almost always leads to significant decrease in prompt sensitivity. We also find that alterations to prompt template lead to the highest sensitivity in the case of MCQtype tasks, whereas paraphrasing results in the highest sensitivity in open-ended generation tasks. The code for reproducing our results is open-sourced at this https URL.

Title: BACKTIME: Backdoor Attacks on Multivariate Time Series Forecasting

• Authors: Xiao Lin, Zhining Liu, Dongqi Fu, Ruizhong Qiu, Hanghang Tong
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02195
• Pdf URL: https://arxiv.org/pdf/2410.02195
• Copy Paste: [[2410.02195]] BACKTIME: Backdoor Attacks on Multivariate Time Series Forecasting(https://arxiv.org/abs/2410.02195)
• Keywords: attack, robust, steal
• Abstract: Multivariate Time Series (MTS) forecasting is a fundamental task with numerous real-world applications, such as transportation, climate, and epidemiology. While a myriad of powerful deep learning models have been developed for this task, few works have explored the robustness of MTS forecasting models to malicious attacks, which is crucial for their trustworthy employment in high-stake scenarios. To address this gap, we dive deep into the backdoor attacks on MTS forecasting models and propose an effective attack method named this http URL subtly injecting a few stealthy triggers into the MTS data, BackTime can alter the predictions of the forecasting model according to the attacker's intent. Specifically, BackTime first identifies vulnerable timestamps in the data for poisoning, and then adaptively synthesizes stealthy and effective triggers by solving a bi-level optimization problem with a GNN-based trigger generator. Extensive experiments across multiple datasets and state-of-the-art MTS forecasting models demonstrate the effectiveness, versatility, and stealthiness of \method{} attacks. The code is available at \url{this https URL}.

Title: G2T-LLM: Graph-to-Tree Text Encoding for Molecule Generation with Fine-Tuned Large Language Models

• Authors: Zhaoning Yu, Xiangyang Xu, Hongyang Gao
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.02198
• Pdf URL: https://arxiv.org/pdf/2410.02198
• Copy Paste: [[2410.02198]] G2T-LLM: Graph-to-Tree Text Encoding for Molecule Generation with Fine-Tuned Large Language Models(https://arxiv.org/abs/2410.02198)
• Keywords: large language model
• Abstract: We introduce G2T-LLM, a novel approach for molecule generation that uses graph-to-tree text encoding to transform graph-based molecular structures into a hierarchical text format optimized for large language models (LLMs). This encoding converts complex molecular graphs into tree-structured formats, such as JSON and XML, which LLMs are particularly adept at processing due to their extensive pre-training on these types of data. By leveraging the flexibility of LLMs, our approach allows for intuitive interaction using natural language prompts, providing a more accessible interface for molecular design. Through supervised fine-tuning, G2T-LLM generates valid and coherent chemical structures, addressing common challenges like invalid outputs seen in traditional graph-based methods. While LLMs are computationally intensive, they offer superior generalization and adaptability, enabling the generation of diverse molecular structures with minimal task-specific customization. The proposed approach achieved comparable performances with state-of-the-art methods on various benchmark molecular generation datasets, demonstrating its potential as a flexible and innovative tool for AI-driven molecular design.

Title: Measuring, Evaluating and Improving Logical Consistency in Large Language Models

• Authors: Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulić, Nigel Collier
• Subjects: cs.CL, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2410.02205
• Pdf URL: https://arxiv.org/pdf/2410.02205
• Copy Paste: [[2410.02205]] Measuring, Evaluating and Improving Logical Consistency in Large Language Models(https://arxiv.org/abs/2410.02205)
• Keywords: robust, large language model
• Abstract: Recent research in Large Language Models (LLMs) has shown promising progress related to LLM alignment with human preferences. LLM-empowered decision-making systems are expected to be predictable, reliable and trustworthy, which implies being free from paradoxes or contradictions that could undermine their credibility and validity. However, LLMs still exhibit inconsistent and biased behaviour when making decisions or judgements. In this work, we focus on studying logical consistency of LLMs as a prerequisite for more reliable and trustworthy systems. Logical consistency ensures that decisions are based on a stable and coherent understanding of the problem, reducing the risk of erratic or contradictory outputs. We first propose a universal framework to quantify the logical consistency via three fundamental proxies: transitivity, commutativity and negation invariance. We then evaluate logical consistency, using the defined measures, of a wide range of LLMs, demonstrating that it can serve as a strong proxy for overall robustness. Additionally, we introduce a data refinement and augmentation technique that enhances the logical consistency of LLMs without sacrificing alignment to human preferences. It augments noisy and sparse pairwise-comparison annotations by estimating a partially or totally ordered preference rankings using rank aggregation methods. Finally, we show that logical consistency impacts the performance of LLM-based logic-dependent algorithms, where LLMs serve as logical operators.

Title: Adapting Segment Anything Model to Melanoma Segmentation in Microscopy Slide Images

• Authors: Qingyuan Liu, Avideh Zakhor
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02207
• Pdf URL: https://arxiv.org/pdf/2410.02207
• Copy Paste: [[2410.02207]] Adapting Segment Anything Model to Melanoma Segmentation in Microscopy Slide Images(https://arxiv.org/abs/2410.02207)
• Keywords: segmentation
• Abstract: Melanoma segmentation in Whole Slide Images (WSIs) is useful for prognosis and the measurement of crucial prognostic factors such as Breslow depth and primary invasive tumor size. In this paper, we present a novel approach that uses the Segment Anything Model (SAM) for automatic melanoma segmentation in microscopy slide images. Our method employs an initial semantic segmentation model to generate preliminary segmentation masks that are then used to prompt SAM. We design a dynamic prompting strategy that uses a combination of centroid and grid prompts to achieve optimal coverage of the super high-resolution slide images while maintaining the quality of generated prompts. To optimize for invasive melanoma segmentation, we further refine the prompt generation process by implementing in-situ melanoma detection and low-confidence region filtering. We select Segformer as the initial segmentation model and EfficientSAM as the segment anything model for parameter-efficient fine-tuning. Our experimental results demonstrate that this approach not only surpasses other state-of-the-art melanoma segmentation methods but also significantly outperforms the baseline Segformer by 9.1% in terms of IoU.

Title: Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference

• Authors: Wei Cheng, Tianlu Wang, Yanmin Ji, Fan Yang, Keren Tan, Yiyu Zheng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02210
• Pdf URL: https://arxiv.org/pdf/2410.02210
• Copy Paste: [[2410.02210]] Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference(https://arxiv.org/abs/2410.02210)
• Keywords: large language model
• Abstract: While in-context learning with large language models (LLMs) has shown impressive performance, we have discovered a unique miscalibration behavior where both correct and incorrect predictions are assigned the same level of confidence. We refer to this phenomenon as indiscriminate miscalibration. We found that traditional calibration metrics, such as Expected Calibrated Errors (ECEs), are unable to capture this behavior effectively. To address this issue, we propose new metrics to measure the severity of indiscriminate miscalibration. Additionally, we develop a novel in-context comparative inference method to alleviate miscalibrations and improve classification performance. Through extensive experiments on five datasets, we demonstrate that our proposed method can achieve more accurate and calibrated predictions compared to regular zero-shot and few-shot prompting.

Title: Buckle Up: Robustifying LLMs at Every Customization Stage via Data Curation

• Authors: Xiaoqun Liu, Jiacheng Liang, Luoxi Tang, Chenyu You, Muchao Ye, Zhaohan Xi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02220
• Pdf URL: https://arxiv.org/pdf/2410.02220
• Copy Paste: [[2410.02220]] Buckle Up: Robustifying LLMs at Every Customization Stage via Data Curation(https://arxiv.org/abs/2410.02220)
• Keywords: secure, attack, robust, large language model
• Abstract: Large language models (LLMs) are extensively adapted for downstream applications through a process known as "customization," with fine-tuning being a common method for integrating domain-specific expertise. However, recent studies have revealed a vulnerability that tuning LLMs with malicious samples can compromise their robustness and amplify harmful content, an attack known as "jailbreaking." To mitigate such attack, we propose an effective defensive framework utilizing data curation to revise commonsense texts and enhance their safety implication from the perspective of LLMs. The curated texts can mitigate jailbreaking attacks at every stage of the customization process: before customization to immunize LLMs against future jailbreak attempts, during customization to neutralize jailbreaking risks, or after customization to restore the compromised models. Since the curated data strengthens LLMs through the standard fine-tuning workflow, we do not introduce additional modules during LLM inference, thereby preserving the original customization process. Experimental results demonstrate a substantial reduction in jailbreaking effects, with up to a 100% success in generating responsible responses. Notably, our method is effective even with commonsense texts, which are often more readily available than safety-relevant data. With the every-stage defensive framework and supporting experimental performance, this work represents a significant advancement in mitigating jailbreaking risks and ensuring the secure customization of LLMs.

Title: EmbedLLM: Learning Compact Representations of Large Language Models

• Authors: Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, Kannan Ramchandran
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02223
• Pdf URL: https://arxiv.org/pdf/2410.02223
• Copy Paste: [[2410.02223]] EmbedLLM: Learning Compact Representations of Large Language Models(https://arxiv.org/abs/2410.02223)
• Keywords: large language model
• Abstract: With hundreds of thousands of language models available on Huggingface today, efficiently evaluating and utilizing these models across various downstream, tasks has become increasingly critical. Many existing methods repeatedly learn task-specific representations of Large Language Models (LLMs), which leads to inefficiencies in both time and computational resources. To address this, we propose EmbedLLM, a framework designed to learn compact vector representations, of LLMs that facilitate downstream applications involving many models, such as model routing. We introduce an encoder-decoder approach for learning such embeddings, along with a systematic framework to evaluate their effectiveness. Empirical results show that EmbedLLM outperforms prior methods in model routing both in accuracy and latency. Additionally, we demonstrate that our method can forecast a model's performance on multiple benchmarks, without incurring additional inference cost. Extensive probing experiments validate that the learned embeddings capture key model characteristics, e.g. whether the model is specialized for coding tasks, even without being explicitly trained on them. We open source our dataset, code and embedder to facilitate further research and application.

Title: Efficient Semantic Segmentation via Lightweight Multiple-Information Interaction Network

• Authors: Yangyang Qiu, Guoan Xu, Guangwei Gao, Zhenhua Guo, Yi Yu, Chia-Wen Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02224
• Pdf URL: https://arxiv.org/pdf/2410.02224
• Copy Paste: [[2410.02224]] Efficient Semantic Segmentation via Lightweight Multiple-Information Interaction Network(https://arxiv.org/abs/2410.02224)
• Keywords: transformer, segmentation
• Abstract: Recently, the integration of the local modeling capabilities of Convolutional Neural Networks (CNNs) with the global dependency strengths of Transformers has created a sensation in the semantic segmentation community. However, substantial computational workloads and high hardware memory demands remain major obstacles to their further application in real-time scenarios. In this work, we propose a lightweight multiple-information interaction network for real-time semantic segmentation, called LMIINet, which effectively combines CNNs and Transformers while reducing redundant computations and memory footprint. It features Lightweight Feature Interaction Bottleneck (LFIB) modules comprising efficient convolutions that enhance context integration. Additionally, improvements are made to the Flatten Transformer by enhancing local and global feature interaction to capture detailed semantic information. The incorporation of a combination coefficient learning scheme in both LFIB and Transformer blocks facilitates improved feature interaction. Extensive experiments demonstrate that LMIINet excels in balancing accuracy and efficiency. With only 0.72M parameters and 11.74G FLOPs, LMIINet achieves 72.0% mIoU at 100 FPS on the Cityscapes test set and 69.94% mIoU at 160 FPS on the CamVid test dataset using a single RTX2080Ti GPU.

Title: Mitigating Downstream Model Risks via Model Provenance

• Authors: Keyu Wang, Abdullah Norozi Iranzad, Scott Schaffter, Doina Precup, Jonathan Lebensold
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02230
• Pdf URL: https://arxiv.org/pdf/2410.02230
• Copy Paste: [[2410.02230]] Mitigating Downstream Model Risks via Model Provenance(https://arxiv.org/abs/2410.02230)
• Keywords: security
• Abstract: Research and industry are rapidly advancing the innovation and adoption of foundation model-based systems, yet the tools for managing these models have not kept pace. Understanding the provenance and lineage of models is critical for researchers, industry, regulators, and public trust. While model cards and system cards were designed to provide transparency, they fall short in key areas: tracing model genealogy, enabling machine readability, offering reliable centralized management systems, and fostering consistent creation incentives. This challenge mirrors issues in software supply chain security, but AI/ML remains at an earlier stage of maturity. Addressing these gaps requires industry-standard tooling that can be adopted by foundation model publishers, open-source model innovators, and major distribution platforms. We propose a machine-readable model specification format to simplify the creation of model records, thereby reducing error-prone human effort, notably when a new model inherits most of its design from a foundation model. Our solution explicitly traces relationships between upstream and downstream models, enhancing transparency and traceability across the model lifecycle. To facilitate the adoption, we introduce the unified model record (UMR) repository , a semantically versioned system that automates the publication of model records to multiple formats (PDF, HTML, LaTeX) and provides a hosted web interface (this https URL). This proof of concept aims to set a new standard for managing foundation models, bridging the gap between innovation and responsible model management.

Title: Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features

• Authors: Chengkai Hou, Zhengrong Xue, Bingyang Zhou, Jinghan Ke, Lin Shao, Huazhe Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02237
• Pdf URL: https://arxiv.org/pdf/2410.02237
• Copy Paste: [[2410.02237]] Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features(https://arxiv.org/abs/2410.02237)
• Keywords: robust
• Abstract: Detecting 3D keypoints with semantic consistency is widely used in many scenarios such as pose estimation, shape registration and robotics. Currently, most unsupervised 3D keypoint detection methods focus on the rigid-body objects. However, when faced with deformable objects, the keypoints they identify do not preserve semantic consistency well. In this paper, we introduce an innovative unsupervised keypoint detector Key-Grid for both the rigid-body and deformable objects, which is an autoencoder framework. The encoder predicts keypoints and the decoder utilizes the generated keypoints to reconstruct the objects. Unlike previous work, we leverage the identified keypoint in formation to form a 3D grid feature heatmap called grid heatmap, which is used in the decoder section. Grid heatmap is a novel concept that represents the latent variables for grid points sampled uniformly in the 3D cubic space, where these variables are the shortest distance between the grid points and the skeleton connected by keypoint pairs. Meanwhile, we incorporate the information from each layer of the encoder into the decoder section. We conduct an extensive evaluation of Key-Grid on a list of benchmark datasets. Key-Grid achieves the state-of-the-art performance on the semantic consistency and position accuracy of keypoints. Moreover, we demonstrate the robustness of Key-Grid to noise and downsampling. In addition, we achieve SE-(3) invariance of keypoints though generalizing Key-Grid to a SE(3)-invariant backbone.

Title: SCA: Highly Efficient Semantic-Consistent Unrestricted Adversarial Attack

• Authors: Zihao Pan, Weibin Wu, Yuhang Cao, Zibin Zheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02240
• Pdf URL: https://arxiv.org/pdf/2410.02240
• Copy Paste: [[2410.02240]] SCA: Highly Efficient Semantic-Consistent Unrestricted Adversarial Attack(https://arxiv.org/abs/2410.02240)
• Keywords: attack, diffusion, large language model
• Abstract: Unrestricted adversarial attacks typically manipulate the semantic content of an image (e.g., color or texture) to create adversarial examples that are both effective and photorealistic. Recent works have utilized the diffusion inversion process to map images into a latent space, where high-level semantics are manipulated by introducing perturbations. However, they often results in substantial semantic distortions in the denoised output and suffers from low efficiency. In this study, we propose a novel framework called Semantic-Consistent Unrestricted Adversarial Attacks (SCA), which employs an inversion method to extract edit-friendly noise maps and utilizes Multimodal Large Language Model (MLLM) to provide semantic guidance throughout the process. Under the condition of rich semantic information provided by MLLM, we perform the DDPM denoising process of each step using a series of edit-friendly noise maps, and leverage DPM Solver++ to accelerate this process, enabling efficient sampling with semantic consistency. Compared to existing methods, our framework enables the efficient generation of adversarial examples that exhibit minimal discernible semantic changes. Consequently, we for the first time introduce Semantic-Consistent Adversarial Examples (SCAE). Extensive experiments and visualizations have demonstrated the high efficiency of SCA, particularly in being on average 12 times faster than the state-of-the-art attacks. Our code can be found at this https URL}{this https URL.

Title: Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis

• Authors: Hyunwoo Lee, Hayoung Choi, Hyunju Kim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02242
• Pdf URL: https://arxiv.org/pdf/2410.02242
• Copy Paste: [[2410.02242]] Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis(https://arxiv.org/abs/2410.02242)
• Keywords: robust
• Abstract: As a neural network's depth increases, it can achieve strong generalization performance. Training, however, becomes challenging due to gradient issues. Theoretical research and various methods have been introduced to address this issues. However, research on weight initialization methods that can be effectively applied to tanh neural networks of varying sizes still needs to be completed. This paper presents a novel weight initialization method for Feedforward Neural Networks with tanh activation function. Based on an analysis of the fixed points of the function $\tanh(ax)$, our proposed method aims to determine values of $a$ that prevent the saturation of activations. A series of experiments on various classification datasets demonstrate that the proposed method is more robust to network size variations than the existing method. Furthermore, when applied to Physics-Informed Neural Networks, the method exhibits faster convergence and robustness to variations of the network size compared to Xavier initialization in problems of Partial Differential Equations.

Title: Visual Prompting in LLMs for Enhancing Emotion Recognition

• Authors: Qixuan Zhang, Zhifeng Wang, Dylan Zhang, Wenjia Niu, Sabrina Caldwell, Tom Gedeon, Yang Liu, Zhenyue Qin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02244
• Pdf URL: https://arxiv.org/pdf/2410.02244
• Copy Paste: [[2410.02244]] Visual Prompting in LLMs for Enhancing Emotion Recognition(https://arxiv.org/abs/2410.02244)
• Keywords: large language model
• Abstract: Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing. Nonetheless, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. To address this problem, we propose a Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through a battery of experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model's ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.

Title: PFGuard: A Generative Framework with Privacy and Fairness Safeguards

• Authors: Soyeon Kim, Yuji Roh, Geon Heo, Steven Euijong Whang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02246
• Pdf URL: https://arxiv.org/pdf/2410.02246
• Copy Paste: [[2410.02246]] PFGuard: A Generative Framework with Privacy and Fairness Safeguards(https://arxiv.org/abs/2410.02246)
• Keywords: privacy, fair, generative
• Abstract: Generative models must ensure both privacy and fairness for Trustworthy AI. While these goals have been pursued separately, recent studies propose to combine existing privacy and fairness techniques to achieve both goals. However, naively combining these techniques can be insufficient due to privacy-fairness conflicts, where a sample in a minority group may be amplified for fairness, only to be suppressed for privacy. We demonstrate how these conflicts lead to adverse effects, such as privacy violations and unexpected fairness-utility tradeoffs. To mitigate these risks, we propose PFGuard, a generative framework with privacy and fairness safeguards, which simultaneously addresses privacy, fairness, and utility. By using an ensemble of multiple teacher models, PFGuard balances privacy-fairness conflicts between fair and private training stages and achieves high utility based on ensemble learning. Extensive experiments show that PFGuard successfully generates synthetic data on high-dimensional data while providing both fairness convergence and strict DP guarantees - the first of its kind to our knowledge.

Title: Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization

• Authors: Xinhao Yao, Hongjin Qian, Xiaolin Hu, Gengze Xu, Yong Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02247
• Pdf URL: https://arxiv.org/pdf/2410.02247
• Copy Paste: [[2410.02247]] Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization(https://arxiv.org/abs/2410.02247)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs), built on Transformer architectures, exhibit remarkable generalization across a wide range of tasks. However, fine-tuning these models for specific tasks remains resource-intensive due to their extensive parameterization. In this paper, we investigate two remarkable phenomena observed during the fine-tuning of LLMs, particularly focusing on the attention mechanism: (1) Different Impact, optimizing the $\mathbf{W}_v$ matrix significantly improves performance over optimizing the $\mathbf{W}_k$ matrix. Fine-tuning only the $\mathbf{W}_q$ and $\mathbf{W}_v$ matrices is computationally efficient, delivering results that are comparable to, or even better than, fine-tuning all three matrices $\mathbf{W}_q$, $\mathbf{W}_k$, and $\mathbf{W}_v$. (2) Efficient Convergence, employing distinct learning rates for these matrices is crucial for optimal performance, with a higher learning rate for the $\mathbf{W}_v$ matrix expediting convergence. However, theoretical analyses of these phenomena are still relatively limited. We present a theoretical analysis of these phenomena from two perspectives: (i) Generalization, where we demonstrate that fine-tuning only $\mathbf{W}_q$ and $\mathbf{W}_v$ improves generalization bounds, enhances memory efficiency, and (ii) Optimization, where we emphasize that the feature learning of the attention mechanism is efficient, particularly when using distinct learning rates for the matrices, which leads to more effective fine-tuning. Building on these insights, we propose a new strategy that improves fine-tuning efficiency in terms of both storage and time. Experimental results on benchmark datasets validate the effectiveness of this approach, supporting our theoretical findings. Our analysis lays the theoretical groundwork for configuring and improving lightweight algorithms in LLMs fine-tuning.

Title: Probabilistic road classification in historical maps using synthetic data and deep learning

• Authors: Dominik J. Mühlematter, Sebastian Schweizer, Chenjing Jiao, Xue Xia, Magnus Heitzler, Lorenz Hurni
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02250
• Pdf URL: https://arxiv.org/pdf/2410.02250
• Copy Paste: [[2410.02250]] Probabilistic road classification in historical maps using synthetic data and deep learning(https://arxiv.org/abs/2410.02250)
• Keywords: extraction, segmentation
• Abstract: Historical maps are invaluable for analyzing long-term changes in transportation and spatial development, offering a rich source of data for evolutionary studies. However, digitizing and classifying road networks from these maps is often expensive and time-consuming, limiting their widespread use. Recent advancements in deep learning have made automatic road extraction from historical maps feasible, yet these methods typically require large amounts of labeled training data. To address this challenge, we introduce a novel framework that integrates deep learning with geoinformation, computer-based painting, and image processing methodologies. This framework enables the extraction and classification of roads from historical maps using only road geometries without needing road class labels for training. The process begins with training of a binary segmentation model to extract road geometries, followed by morphological operations, skeletonization, vectorization, and filtering algorithms. Synthetic training data is then generated by a painting function that artificially re-paints road segments using predefined symbology for road classes. Using this synthetic data, a deep ensemble is trained to generate pixel-wise probabilities for road classes to mitigate distribution shift. These predictions are then discretized along the extracted road geometries. Subsequently, further processing is employed to classify entire roads, enabling the identification of potential changes in road classes and resulting in a labeled road class dataset. Our method achieved completeness and correctness scores of over 94% and 92%, respectively, for road class 2, the most prevalent class in the two Siegfried Map sheets from Switzerland used for testing. This research offers a powerful tool for urban planning and transportation decision-making by efficiently extracting and classifying roads from historical maps.

Title: Alignment of Cybersecurity Incident Prioritisation with Incident Response Management Maturity Capabilities

• Authors: Abdulaziz Gulay, Leandros Maglaras
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02259
• Pdf URL: https://arxiv.org/pdf/2410.02259
• Copy Paste: [[2410.02259]] Alignment of Cybersecurity Incident Prioritisation with Incident Response Management Maturity Capabilities(https://arxiv.org/abs/2410.02259)
• Keywords: security, robust
• Abstract: The increasing frequency and sophistication of cybersecurity incidents pose significant challenges to organisations, highlighting the critical need for robust incident response capabilities. This paper explores a possible utilisation of IR CMMs assessments to systematically prioritise incidents based on their impact, severity, and the incident response capabilities of an organisation in specific areas associated with human and organisational factors. The findings reveal common weaknesses in incident response, such as inadequate training and poor communication, and highlight best practices, including regular training programs, clear communication protocols, and well-documented response procedures. The analysis also emphasises the importance of organisational culture in enhancing incident response capabilities. By addressing the gap in understanding how the output of IRM assessments can be immediately utilised to prioritise high-risk incidents, this paper contributes valuable insights to academia and practice, offering a structured approach to enhancing organisational resilience against cybersecurity threats.

Title: FedScalar: A Communication efficient Federated Learning

• Authors: M. Rostami, S. S. Kia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02260
• Pdf URL: https://arxiv.org/pdf/2410.02260
• Copy Paste: [[2410.02260]] FedScalar: A Communication efficient Federated Learning(https://arxiv.org/abs/2410.02260)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) has gained considerable popularity for distributed machine learning due to its ability to preserve the privacy of participating agents by eliminating the need for data aggregation. Nevertheless, communication costs between agents and the central server in FL are substantial in large-scale problems and remain a limiting factor for this algorithm. This paper introduces an innovative algorithm, called \emph{FedScalar}, within the federated learning framework aimed at improving communication efficiency. Unlike traditional FL methods that require agents to send high-dimensional vectors to the server, \emph{FedScalar} enables agents to communicate updates using a single scalar. Each agent encodes its updated model parameters into a scalar through the inner product between its local update difference and a random vector, which is then transmitted to the server. The server decodes this information by projecting the averaged scalar values onto the random vector. Our method thereby significantly reduces communication overhead. Technically, we demonstrate that the proposed algorithm achieves a convergence rate of $O(1/\sqrt{K})$ to a stationary point for smooth, non-convex loss functions. Additionally, our analysis shows that altering the underlying distribution of the random vector generated by the server can reduce the variance during the aggregation step of the algorithm. Finally, we validate the performance and communication efficiency of our algorithm with numerical simulations.

Title: Unsupervised Meta-Learning via Dynamic Head and Heterogeneous Task Construction for Few-Shot Classification

• Authors: Yunchuan Guan, Yu Liu, Ketong Liu, Ke Zhou, Zhiqi Shen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02267
• Pdf URL: https://arxiv.org/pdf/2410.02267
• Copy Paste: [[2410.02267]] Unsupervised Meta-Learning via Dynamic Head and Heterogeneous Task Construction for Few-Shot Classification(https://arxiv.org/abs/2410.02267)
• Keywords: robust
• Abstract: Meta-learning has been widely used in recent years in areas such as few-shot learning and reinforcement learning. However, the questions of why and when it is better than other algorithms in few-shot classification remain to be explored. In this paper, we perform pre-experiments by adjusting the proportion of label noise and the degree of task heterogeneity in the dataset. We use the metric of Singular Vector Canonical Correlation Analysis to quantify the representation stability of the neural network and thus to compare the behavior of meta-learning and classical learning algorithms. We find that benefiting from the bi-level optimization strategy, the meta-learning algorithm has better robustness to label noise and heterogeneous tasks. Based on the above conclusion, we argue a promising future for meta-learning in the unsupervised area, and thus propose DHM-UHT, a dynamic head meta-learning algorithm with unsupervised heterogeneous task construction. The core idea of DHM-UHT is to use DBSCAN and dynamic head to achieve heterogeneous task construction and meta-learn the whole process of unsupervised heterogeneous task construction. On several unsupervised zero-shot and few-shot datasets, DHM-UHT obtains state-of-the-art performance. The code is released at this https URL.

Title: Perfect Counterfactuals in Imperfect Worlds: Modelling Noisy Implementation of Actions in Sequential Algorithmic Recourse

• Authors: Yueqing Xuan, Kacper Sokol, Mark Sanderson, Jeffrey Chan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02273
• Pdf URL: https://arxiv.org/pdf/2410.02273
• Copy Paste: [[2410.02273]] Perfect Counterfactuals in Imperfect Worlds: Modelling Noisy Implementation of Actions in Sequential Algorithmic Recourse(https://arxiv.org/abs/2410.02273)
• Keywords: robust
• Abstract: Algorithmic recourse provides actions to individuals who have been adversely affected by automated decision-making and helps them achieve a desired outcome. Knowing the recourse, however, does not guarantee that users would implement it perfectly, either due to environmental variability or personal choices. Recourse generation should thus anticipate its sub-optimal or noisy implementation. While several approaches have constructed recourse that accounts for robustness to small perturbation (i.e., noisy recourse implementation), they assume an entire recourse to be implemented in a single step and thus apply one-off uniform noise to it. Such assumption is unrealistic since recourse often includes multiple sequential steps which becomes harder to implement and subject to more noise. In this work, we consider recourse under plausible noise that adapts to the local data geometry and accumulates at every step of the way. We frame this problem as a Markov Decision Process and demonstrate that the distribution of our plausible noise satisfies the Markov property. We then propose the RObust SEquential (ROSE) recourse generator to output a sequence of steps that will lead to the desired outcome even under imperfect implementation. Given our plausible modelling of sub-optimal human actions and greater recourse robustness to accumulated uncertainty, ROSE can grant users higher chances of success under low recourse costs. Empirical evaluation shows our algorithm manages the inherent trade-off between recourse robustness and costs more effectively while ensuring its low sparsity and fast computation.

Title: Morphological evaluation of subwords vocabulary used by BETO language model

• Authors: Óscar García-Sierra, Ana Fernández-Pampillón Cesteros, Miguel Ortega-Martín
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02283
• Pdf URL: https://arxiv.org/pdf/2410.02283
• Copy Paste: [[2410.02283]] Morphological evaluation of subwords vocabulary used by BETO language model(https://arxiv.org/abs/2410.02283)
• Keywords: large language model
• Abstract: Subword tokenization algorithms used by Large Language Models are significantly more efficient and can independently build the necessary vocabulary of words and subwords without human intervention. However, those subwords do not always align with real morphemes, potentially impacting the models' performance, though it remains uncertain when this might occur. In previous research, we proposed a method to assess the morphological quality of vocabularies, focusing on the overlap between these vocabularies and the morphemes of a given language. Our evaluation method was built on three quality measures, relevance, cohesion, and morphological accuracy, and a procedure for their assessment. By applying this method to vocabularies created by three subword tokenization algorithms, BPE, Wordpiece, and Unigram, we concluded that these vocabularies generally exhibit very low morphological quality. In this article, we apply this evaluation to the tokenizer of BETO, a BERT language model trained on large Spanish corpora. This evaluation, along with our previous results, helped us conclude that its vocabulary has a low morphological quality, and we also found that training the tokenizer in a larger corpus does not improve the morphological quality of the generated vocabulary. Additionally, this evaluation helps clarify the algorithm used by the tokenizer, that is, Wordpiece, given the inconsistencies between the authors' claims and the model's configuration.

Title: Correlation and Navigation in the Vocabulary Key Representation Space of Language Models

• Authors: Letian Peng, Chenyang An, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02284
• Pdf URL: https://arxiv.org/pdf/2410.02284
• Copy Paste: [[2410.02284]] Correlation and Navigation in the Vocabulary Key Representation Space of Language Models(https://arxiv.org/abs/2410.02284)
• Keywords: transformer
• Abstract: Language model (LM) decoding is based on the next-token prediction (NTP) probability distribution. For neural LMs (e.g., Transformer-based), NTP distribution is essentially a softmax-regularized dot product between an encoded input context (query) and fixed vocabulary representations (keys). In this paper, we study the effect of the key distribution on the NTP distribution, with a focus on whether the similarity between keys will trigger spurious correlations in NTP. Through knowledge-probing tasks, we show that in the NTP distribution, the few top-ranked tokens are typically accurate. However, the middle-ranked prediction is highly biased towards the tokens that are distributionally (not necessarily semantically) similar to these top ones. For instance, if "P" is predicted as the top-1 token, "A"-"Z" will all be ranked high in NTP, no matter whether they can lead to correct decoding results. This hurts the sampling diversity and makes the sampling of correct, long-tail results hopeless and noisy. We attempt to alleviate this issue via a novel in-context method that iteratively pushes the query representation away from explored regions. Specifically, we include the explored decoding results in the context and prompt the LM to generate something else, which encourages the LM to produce a query representation that has small dot products with explored keys. Experiments on knowledge-probing tasks show that our method leads to efficient navigation away from explored keys to correct new keys. We further extend our method to open-ended and chain-of-thought (for reasoning) generation. Experiment results show that ICN contributes to better generation diversity and improved self-consistency voting performance. Finally, we discuss potential training issues caused by the fixed key space together with the challenges and possible ways to address them in future research.

Title: Efficient Second-Order Neural Network Optimization via Adaptive Trust Region Methods

• Authors: James Vo
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02293
• Pdf URL: https://arxiv.org/pdf/2410.02293
• Copy Paste: [[2410.02293]] Efficient Second-Order Neural Network Optimization via Adaptive Trust Region Methods(https://arxiv.org/abs/2410.02293)
• Keywords: robust, large language model
• Abstract: Second-order optimization methods offer notable advantages in training deep neural networks by utilizing curvature information to achieve faster convergence. However, traditional second-order techniques are computationally prohibitive, primarily due to the large matrix inversions and high memory demands they require. While adaptive trust-region methods have been developed to mitigate these issues, their performance is often hindered by conservative estimates of key parameters, such as the Lipschitz constant of the Hessian, resulting in suboptimal outcomes. In this paper, we introduce SecondOrderAdaptiveAdam (SOAA), a novel optimization algorithm designed to overcome these limitations. SOAA approximates the Fisher information matrix using a diagonal representation, reducing computational complexity from \(O(n^{2})\) to \(O(n)\), thereby making it suitable for large-scale deep learning models, including large language models (LLMs). Additionally, the algorithm integrates an adaptive trust-region mechanism that dynamically adjusts the trust region size based on observed loss reduction, ensuring both robust convergence and computational efficiency. We empirically demonstrate that SOAA achieves faster and more stable convergence compared to first-order optimizers, such as Adam, under similar computational constraints. However, the diagonal approximation of the Fisher information matrix may be less effective in capturing higher-order interactions between gradients, suggesting potential areas for further refinement and future research.

Title: Language Models are Graph Learners

• Authors: Zhe Xu, Kaveh Hassani, Si Zhang, Hanqing Zeng, Michihiro Yasunaga, Limei Wang, Dongqi Fu, Ning Yao, Bo Long, Hanghang Tong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02296
• Pdf URL: https://arxiv.org/pdf/2410.02296
• Copy Paste: [[2410.02296]] Language Models are Graph Learners(https://arxiv.org/abs/2410.02296)
• Keywords: transformer
• Abstract: Language Models (LMs) are increasingly challenging the dominance of domain-specific models, including Graph Neural Networks (GNNs) and Graph Transformers (GTs), in graph learning tasks. Following this trend, we propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art GNNs on node classification tasks, without requiring any architectural modification. By preserving the LM's original architecture, our approach retains a key benefit of LM instruction tuning: the ability to jointly train on diverse datasets, fostering greater flexibility and efficiency. To achieve this, we introduce two key augmentation strategies: (1) Enriching LMs' input using topological and semantic retrieval methods, which provide richer contextual information, and (2) guiding the LMs' classification process through a lightweight GNN classifier that effectively prunes class candidates. Our experiments on real-world datasets show that backbone Flan-T5 models equipped with these augmentation strategies outperform state-of-the-art text-output node classifiers and are comparable to top-performing vector-output node classifiers. By bridging the gap between specialized task-specific node classifiers and general LMs, this work paves the way for more versatile and widely applicable graph learning models. We will open-source the code upon publication.

Title: Make Compound Sentences Simple to Analyze: Learning to Split Sentences for Aspect-based Sentiment Analysis

• Authors: Yongsik Seo, Sungwon Song, Ryang Heo, Jieyong Kim, Dongha Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02297
• Pdf URL: https://arxiv.org/pdf/2410.02297
• Copy Paste: [[2410.02297]] Make Compound Sentences Simple to Analyze: Learning to Split Sentences for Aspect-based Sentiment Analysis(https://arxiv.org/abs/2410.02297)
• Keywords: extraction, generative
• Abstract: In the domain of Aspect-Based Sentiment Analysis (ABSA), generative methods have shown promising results and achieved substantial advancements. However, despite these advancements, the tasks of extracting sentiment quadruplets, which capture the nuanced sentiment expressions within a sentence, remain significant challenges. In particular, compound sentences can potentially contain multiple quadruplets, making the extraction task increasingly difficult as sentence complexity grows. To address this issue, we are focusing on simplifying sentence structures to facilitate the easier recognition of these elements and crafting a model that integrates seamlessly with various ABSA tasks. In this paper, we propose Aspect Term Oriented Sentence Splitter (ATOSS), which simplifies compound sentence into simpler and clearer forms, thereby clarifying their structure and intent. As a plug-and-play module, this approach retains the parameters of the ABSA model while making it easier to identify essential intent within input sentences. Extensive experimental results show that utilizing ATOSS outperforms existing methods in both ASQP and ACOS tasks, which are the primary tasks for extracting sentiment quadruplets.

Title: Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models

• Authors: Guobin Shen, Dongcheng Zhao, Yiting Dong, Xiang He, Yi Zeng
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02298
• Pdf URL: https://arxiv.org/pdf/2410.02298
• Copy Paste: [[2410.02298]] Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models(https://arxiv.org/abs/2410.02298)
• Keywords: defense, attack, large language model
• Abstract: As large language models (LLMs) become integral to various applications, ensuring both their safety and utility is paramount. Jailbreak attacks, which manipulate LLMs into generating harmful content, pose significant challenges to this balance. Existing defenses, such as prompt engineering and safety fine-tuning, often introduce computational overhead, increase inference latency, and lack runtime flexibility. Moreover, overly restrictive safety measures can degrade model utility by causing refusals of benign queries. In this paper, we introduce Jailbreak Antidote, a method that enables real-time adjustment of LLM safety preferences by manipulating a sparse subset of the model's internal states during inference. By shifting the model's hidden representations along a safety direction with varying strengths, we achieve flexible control over the safety-utility balance without additional token overhead or inference delays. Our analysis reveals that safety-related information in LLMs is sparsely distributed; adjusting approximately 5% of the internal state is as effective as modifying the entire state. Extensive experiments on nine LLMs (ranging from 2 billion to 72 billion parameters), evaluated against ten jailbreak attack methods and compared with six defense strategies, validate the effectiveness and efficiency of our approach. By directly manipulating internal states during reasoning, Jailbreak Antidote offers a lightweight, scalable solution that enhances LLM safety while preserving utility, opening new possibilities for real-time safety mechanisms in widely-deployed AI systems.

Title: A Novel Method for Accurate & Real-time Food Classification: The Synergistic Integration of EfficientNetB7, CBAM, Transfer Learning, and Data Augmentation

• Authors: Shayan Rokhva, Babak Teimourpour
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02304
• Pdf URL: https://arxiv.org/pdf/2410.02304
• Copy Paste: [[2410.02304]] A Novel Method for Accurate & Real-time Food Classification: The Synergistic Integration of EfficientNetB7, CBAM, Transfer Learning, and Data Augmentation(https://arxiv.org/abs/2410.02304)
• Keywords: robust
• Abstract: Integrating artificial intelligence into modern society is profoundly transformative, significantly enhancing productivity by streamlining various daily tasks. AI-driven recognition systems provide notable advantages in the food sector, including improved nutrient tracking, tackling food waste, and boosting food production and consumption efficiency. Accurate food classification is a crucial initial step in utilizing advanced AI models, as the effectiveness of this process directly influences the success of subsequent operations; therefore, achieving high accuracy at a reasonable speed is essential. Despite existing research efforts, a gap persists in improving performance while ensuring rapid processing times, prompting researchers to pursue cost-effective and precise models. This study addresses this gap by employing the state-of-the-art EfficientNetB7 architecture, enhanced through transfer learning, data augmentation, and the CBAM attention module. This methodology results in a robust model that surpasses previous studies in accuracy while maintaining rapid processing suitable for real-world applications. The Food11 dataset from Kaggle was utilized, comprising 16643 imbalanced images across 11 diverse classes with significant intra-category diversities and inter-category similarities. Furthermore, the proposed methodology, bolstered by various deep learning techniques, consistently achieves an impressive average accuracy of 96.40%. Notably, it can classify over 60 images within one second during inference on unseen data, demonstrating its ability to deliver high accuracy promptly. This underscores its potential for practical applications in accurate food classification and enhancing efficiency in subsequent processes.

Title: Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models

• Authors: Rui Meng, Ye Liu, Lifu Tu, Daqing He, Yingbo Zhou, Semih Yavuz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02308
• Pdf URL: https://arxiv.org/pdf/2410.02308
• Copy Paste: [[2410.02308]] Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models(https://arxiv.org/abs/2410.02308)
• Keywords: large language model
• Abstract: Phrases are fundamental linguistic units through which humans convey semantics. This study critically examines the capacity of API-based large language models (LLMs) to comprehend phrase semantics, utilizing three human-annotated datasets. We assess the performance of LLMs in executing phrase semantic reasoning tasks guided by natural language instructions and explore the impact of common prompting techniques, including few-shot demonstrations and Chain-of-Thought reasoning. Our findings reveal that LLMs greatly outperform traditional embedding methods across the datasets; however, they do not show a significant advantage over fine-tuned methods. The effectiveness of advanced prompting strategies shows variability. We conduct detailed error analyses to interpret the limitations faced by LLMs in comprehending phrase semantics. Code and data can be found at this https URL.

Title: Decoupling Layout from Glyph in Online Chinese Handwriting Generation

• Authors: Ren-Min Si, Yan-Ming Zhang, Yi Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02309
• Pdf URL: https://arxiv.org/pdf/2410.02309
• Copy Paste: [[2410.02309]] Decoupling Layout from Glyph in Online Chinese Handwriting Generation(https://arxiv.org/abs/2410.02309)
• Keywords: diffusion
• Abstract: Text plays a crucial role in the transmission of human civilization, and teaching machines to generate online handwritten text in various styles presents an interesting and significant challenge. However, most prior work has concentrated on generating individual Chinese fonts, leaving {complete text line generation largely unexplored}. In this paper, we identify that text lines can naturally be divided into two components: layout and glyphs. Based on this division, we designed a text line layout generator coupled with a diffusion-based stylized font synthesizer to address this challenge hierarchically. More concretely, the layout generator performs in-context-like learning based on the text content and the provided style references to generate positions for each glyph autoregressively. Meanwhile, the font synthesizer which consists of a character embedding dictionary, a multi-scale calligraphy style encoder, and a 1D U-Net based diffusion denoiser will generate each font on its position while imitating the calligraphy style extracted from the given style references. Qualitative and quantitative experiments on the CASIA-OLHWDB demonstrate that our method is capable of generating structurally correct and indistinguishable imitation samples.

Title: CTARR: A fast and robust method for identifying anatomical regions on CT images via atlas registration

• Authors: Thomas Buddenkotte, Roland Opfer, Julia Krüger, Alessa Hering, Mireia Crispin-Ortuzar
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02316
• Pdf URL: https://arxiv.org/pdf/2410.02316
• Copy Paste: [[2410.02316]] CTARR: A fast and robust method for identifying anatomical regions on CT images via atlas registration(https://arxiv.org/abs/2410.02316)
• Keywords: robust, segmentation
• Abstract: Medical image analysis tasks often focus on regions or structures located in a particular location within the patient's body. Often large parts of the image may not be of interest for the image analysis task. When using deep-learning based approaches, this causes an unnecessary increases the computational burden during inference and raises the chance of errors. In this paper, we introduce CTARR, a novel generic method for CT Anatomical Region Recognition. The method serves as a pre-processing step for any deep learning-based CT image analysis pipeline by automatically identifying the pre-defined anatomical region that is relevant for the follow-up task and removing the rest. It can be used in (i) image segmentation to prevent false positives in anatomically implausible regions and speeding up the inference, (ii) image classification to produce image crops that are consistent in their anatomical context, and (iii) image registration by serving as a fast pre-registration step. Our proposed method is based on atlas registration and provides a fast and robust way to crop any anatomical region encoded as one or multiple bounding box(es) from any unlabeled CT scan of the brain, chest, abdomen and/or pelvis. We demonstrate the utility and robustness of the proposed method in the context of medical image segmentation by evaluating it on six datasets of public segmentation challenges. The foreground voxels in the regions of interest are preserved in the vast majority of cases and tasks (97.45-100%) while taking only fractions of a seconds to compute (0.1-0.21s) on a deep learning workstation and greatly reducing the segmentation runtime (2.0-12.7x). Our code is available at this https URL.

Title: Post-edits Are Preferences Too

• Authors: Nathaniel Berger, Stefan Riezler, Miriam Exel, Matthias Huck
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02320
• Pdf URL: https://arxiv.org/pdf/2410.02320
• Copy Paste: [[2410.02320]] Post-edits Are Preferences Too(https://arxiv.org/abs/2410.02320)
• Keywords: large language model
• Abstract: Preference Optimization (PO) techniques are currently one of the state of the art techniques for fine-tuning large language models (LLMs) on pairwise preference feedback from human annotators. However, in machine translation, this sort of feedback can be difficult to solicit. Additionally, Kreutzer et al. (2018) have shown that, for machine translation, pairwise preferences are less reliable than other forms of human feedback, such as 5-point ratings. We examine post-edits to see if they can be a source of reliable human preferences by construction. In PO, a human annotator is shown sequences $s_1$ and $s_2$ and asked for a preference judgment, %$s_1 > s_2$; while for post-editing, editors \emph{create} $s_1$ and know that it should be better than $s_2$. We attempt to use these implicit preferences for PO and show that it helps the model move towards post-edit-like hypotheses and away from machine translation-like hypotheses. Furthermore, we show that best results are obtained by pre-training the model with supervised fine-tuning (SFT) on post-edits in order to promote post-edit-like hypotheses to the top output ranks.

Title: Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis

• Authors: Zikun Zhang, Zixiang Chen, Quanquan Gu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02321
• Pdf URL: https://arxiv.org/pdf/2410.02321
• Copy Paste: [[2410.02321]] Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis(https://arxiv.org/abs/2410.02321)
• Keywords: diffusion
• Abstract: Diffusion models have achieved great success in generating high-dimensional samples across various applications. While the theoretical guarantees for continuous-state diffusion models have been extensively studied, the convergence analysis of the discrete-state counterparts remains under-explored. In this paper, we study the theoretical aspects of score-based discrete diffusion models under the Continuous Time Markov Chain (CTMC) framework. We introduce a discrete-time sampling algorithm in the general state space $[S]^d$ that utilizes score estimators at predefined time points. We derive convergence bounds for the Kullback-Leibler (KL) divergence and total variation (TV) distance between the generated sample distribution and the data distribution, considering both scenarios with and without early stopping under specific assumptions. Notably, our KL divergence bounds are nearly linear in dimension $d$, aligning with state-of-the-art results for diffusion models. Our convergence analysis employs a Girsanov-based method and establishes key properties of the discrete score function, which are essential for characterizing the discrete-time sampling process.

Title: RESSCAL3D++: Joint Acquisition and Semantic Segmentation of 3D Point Clouds

• Authors: Remco Royen, Kostas Pataridis, Ward van der Tempel, Adrian Munteanu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02323
• Pdf URL: https://arxiv.org/pdf/2410.02323
• Copy Paste: [[2410.02323]] RESSCAL3D++: Joint Acquisition and Semantic Segmentation of 3D Point Clouds(https://arxiv.org/abs/2410.02323)
• Keywords: segmentation
• Abstract: 3D scene understanding is crucial for facilitating seamless interaction between digital devices and the physical world. Real-time capturing and processing of the 3D scene are essential for achieving this seamless integration. While existing approaches typically separate acquisition and processing for each frame, the advent of resolution-scalable 3D sensors offers an opportunity to overcome this paradigm and fully leverage the otherwise wasted acquisition time to initiate processing. In this study, we introduce VX-S3DIS, a novel point cloud dataset accurately simulating the behavior of a resolution-scalable 3D sensor. Additionally, we present RESSCAL3D++, an important improvement over our prior work, RESSCAL3D, by incorporating an update module and processing strategy. By applying our method to the new dataset, we practically demonstrate the potential of joint acquisition and semantic segmentation of 3D point clouds. Our resolution-scalable approach significantly reduces scalability costs from 2% to just 0.2% in mIoU while achieving impressive speed-ups of 15.6 to 63.9% compared to the non-scalable baseline. Furthermore, our scalable approach enables early predictions, with the first one occurring after only 7% of the total inference time of the baseline. The new VX-S3DIS dataset is available at this https URL.

Title: Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection

• Authors: Tianxiang Chen, Zhentao Tan, Tao Gong, Yue Wu, Qi Chu, Bin Liu, Jieping Ye, Nenghai Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02330
• Pdf URL: https://arxiv.org/pdf/2410.02330
• Copy Paste: [[2410.02330]] Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection(https://arxiv.org/abs/2410.02330)
• Keywords: large language model
• Abstract: As a manner to augment pre-trained large language models (LLM), knowledge injection is critical to develop vertical domain large models and has been widely studied. Although most current approaches, including parameter-efficient fine-tuning (PEFT) and block expansion methods, uniformly apply knowledge across all LLM layers, it raises the question: are all layers equally crucial for knowledge injection? We begin by evaluating the importance of each layer in finding the optimal layer range for knowledge injection. Intuitively, the more important layers should play a more critical role in knowledge injection and deserve a denser injection. We observe performance dips in question-answering benchmarks after the removal or expansion of the shallow layers, and the degradation shrinks as the layer gets deeper, indicating that the shallow layers hold the key to knowledge injection. This insight leads us to propose the S strategy, a post-pretraining strategy of selectively enhancing shallow layers while pruning the less effective deep ones. Based on this strategy, we introduce Llama Slayer-8B and Llama Slayer-8B-Instruct. We experimented on the corpus of code $\&$ math and demonstrated the effectiveness of our strategy. Further experiments across different LLM, Mistral-7B, and a legal corpus confirmed the general applicability of the approach, underscoring its wide-ranging efficacy. Our code is available at: \this https URL

Title: Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks

• Authors: Junlin Hou, Sicen Liu, Yequan Bie, Hongmei Wang, Andong Tan, Luyang Luo, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02331
• Pdf URL: https://arxiv.org/pdf/2410.02331
• Copy Paste: [[2410.02331]] Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks(https://arxiv.org/abs/2410.02331)
• Keywords: robust, explainability
• Abstract: The increasing demand for transparent and reliable models, particularly in high-stakes decision-making areas such as medical image analysis, has led to the emergence of eXplainable Artificial Intelligence (XAI). Post-hoc XAI techniques, which aim to explain black-box models after training, have been controversial in recent works concerning their fidelity to the models' predictions. In contrast, Self-eXplainable AI (S-XAI) offers a compelling alternative by incorporating explainability directly into the training process of deep learning models. This approach allows models to generate inherent explanations that are closely aligned with their internal decision-making processes. Such enhanced transparency significantly supports the trustworthiness, robustness, and accountability of AI systems in real-world medical applications. To facilitate the development of S-XAI methods for medical image analysis, this survey presents an comprehensive review across various image modalities and clinical applications. It covers more than 200 papers from three key perspectives: 1) input explainability through the integration of explainable feature engineering and knowledge graph, 2) model explainability via attention-based learning, concept-based learning, and prototype-based learning, and 3) output explainability by providing counterfactual explanation and textual explanation. Additionally, this paper outlines the desired characteristics of explainability and existing evaluation methods for assessing explanation quality. Finally, it discusses the major challenges and future research directions in developing S-XAI for medical image analysis.

Title: How Much Can RAG Help the Reasoning of LLM?

• Authors: Jingyu Liu, Jiaen Lin, Yong Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02338
• Pdf URL: https://arxiv.org/pdf/2410.02338
• Copy Paste: [[2410.02338]] How Much Can RAG Help the Reasoning of LLM?(https://arxiv.org/abs/2410.02338)
• Keywords: transformer, large language model
• Abstract: Retrieval-Augmented Generation (RAG) has gained significant popularity in modern Large Language Models (LLMs) due to its effectiveness in introducing new knowledge and reducing hallucinations. However, the deep understanding of RAG remains limited, how does RAG help the reasoning process and can RAG help improve the reasoning capability remains question. While external documents are typically considered as a method to incorporate domain-specific information, they also contain intermediate reasoning results related to the query, this suggests that documents could enhance the reasoning capability of LLMs, which has not been previously explored. In this paper, we investigate this issue in depth and find that while RAG can assist with reasoning, the help is limited. If we conceptualize the reasoning process as a tree with fixed depth, then RAG struggles to assist LLMs in performing deeper reasoning. Additionally, the information in the documents requires preprocessing to filter out noise. We demonstrate that this preprocessing is difficult to achieve simply fine-tuning of the LLM, it often necessitates numerous additional transformer layers to solve the problem. To simplify the problem, we propose DPrompt tuning, which effectively resolves the issue within just limited transformer layers, leading to improved performance.

Title: Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA

• Authors: Eduard Tulchinskii, Laida Kushnareva, Kristian Kuznetsov, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02343
• Pdf URL: https://arxiv.org/pdf/2410.02343
• Copy Paste: [[2410.02343]] Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA(https://arxiv.org/abs/2410.02343)
• Keywords: extraction
• Abstract: A standard way to evaluate the abilities of LLM involves presenting a multiple-choice question and selecting the option with the highest logit as the model's predicted answer. However, such a format for evaluating LLMs has limitations, since even if the model knows the correct answer, it may struggle to select the corresponding letter simply due to difficulties in following this rigid format. To address this, we introduce new scores that better capture and reveal model's underlying knowledge: the Query-Key Score (QK-score), derived from the interaction between query and key representations in attention heads, and the Attention Score, based on attention weights. These scores are extracted from specific \textit{select-and-copy} heads, which show consistent performance across popular Multi-Choice Question Answering (MCQA) datasets. Based on these scores, our method improves knowledge extraction, yielding up to 16\% gain for LLaMA2-7B and up to 10\% for larger models on popular MCQA benchmarks. At the same time, the accuracy on a simple synthetic dataset, where the model explicitly knows the right answer, increases by almost 60\%, achieving nearly perfect accuracy, therefore demonstrating the method's efficiency in mitigating MCQA format limitations. To support our claims, we conduct experiments on models ranging from 7 billion to 70 billion parameters in both zero- and few-shot setups.

Title: RelChaNet: Neural Network Feature Selection using Relative Change Scores

• Authors: Felix Zimmer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02344
• Pdf URL: https://arxiv.org/pdf/2410.02344
• Copy Paste: [[2410.02344]] RelChaNet: Neural Network Feature Selection using Relative Change Scores(https://arxiv.org/abs/2410.02344)
• Keywords: interpretability
• Abstract: There is an ongoing effort to develop feature selection algorithms to improve interpretability, reduce computational resources, and minimize overfitting in predictive models. Neural networks stand out as architectures on which to build feature selection methods, and recently, neuron pruning and regrowth have emerged from the sparse neural network literature as promising new tools. We introduce RelChaNet, a novel and lightweight feature selection algorithm that uses neuron pruning and regrowth in the input layer of a dense neural network. For neuron pruning, a gradient sum metric measures the relative change induced in a network after a feature enters, while neurons are randomly regrown. We also propose an extension that adapts the size of the input layer at runtime. Extensive experiments on nine different datasets show that our approach generally outperforms the current state-of-the-art methods, and in particular improves the average accuracy by 2% on the MNIST dataset. Our code is available at this https URL.

Title: Simplicity bias and optimization threshold in two-layer ReLU networks

• Authors: Etienne Boursier, Nicolas Flammarion
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02348
• Pdf URL: https://arxiv.org/pdf/2410.02348
• Copy Paste: [[2410.02348]] Simplicity bias and optimization threshold in two-layer ReLU networks(https://arxiv.org/abs/2410.02348)
• Keywords: diffusion
• Abstract: Understanding generalization of overparametrized neural networks remains a fundamental challenge in machine learning. Most of the literature mostly studies generalization from an interpolation point of view, taking convergence of parameters towards a global minimum of the training loss for granted. While overparametrized architectures indeed interpolated the data for typical classification tasks, this interpolation paradigm does not seem valid anymore for more complex tasks such as in-context learning or diffusion. Instead for such tasks, it has been empirically observed that the trained models goes from global minima to spurious local minima of the training loss as the number of training samples becomes larger than some level we call optimization threshold. While the former yields a poor generalization to the true population loss, the latter was observed to actually correspond to the minimiser of this true loss. This paper explores theoretically this phenomenon in the context of two-layer ReLU networks. We demonstrate that, despite overparametrization, networks often converge toward simpler solutions rather than interpolating the training data, which can lead to a drastic improvement on the test loss with respect to interpolating solutions. Our analysis relies on the so called early alignment phase, during which neurons align towards specific directions. This directional alignment, which occurs in the early stage of training, leads to a simplicity bias, wherein the network approximates the ground truth model without converging to the global minimum of the training loss. Our results suggest that this bias, resulting in an optimization threshold from which interpolation is not reached anymore, is beneficial and enhances the generalization of trained models.

Title: ProtoSeg: A Prototype-Based Point Cloud Instance Segmentation Method

• Authors: Remco Royen, Leon Denis, Adrian Munteanu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02352
• Pdf URL: https://arxiv.org/pdf/2410.02352
• Copy Paste: [[2410.02352]] ProtoSeg: A Prototype-Based Point Cloud Instance Segmentation Method(https://arxiv.org/abs/2410.02352)
• Keywords: segmentation
• Abstract: 3D instance segmentation is crucial for obtaining an understanding of a point cloud scene. This paper presents a novel neural network architecture for performing instance segmentation on 3D point clouds. We propose to jointly learn coefficients and prototypes in parallel which can be combined to obtain the instance predictions. The coefficients are computed using an overcomplete set of sampled points with a novel multi-scale module, dubbed dilated point inception. As the set of obtained instance mask predictions is overcomplete, we employ a non-maximum suppression algorithm to retrieve the final predictions. This approach allows to omit the time-expensive clustering step and leads to a more stable inference time. The proposed method is not only 28% faster than the state-of-the-art, it also exhibits the lowest standard deviation. Our experiments have shown that the standard deviation of the inference time is only 1.0% of the total time while it ranges between 10.8 and 53.1% for the state-of-the-art methods. Lastly, our method outperforms the state-of-the-art both on S3DIS-blocks (4.9% in mRec on Fold-5) and PartNet (2.0% on average in mAP).

Title: AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models

• Authors: Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Xiang Wang, Xiangnan He, Tat-seng Chua
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02355
• Pdf URL: https://arxiv.org/pdf/2410.02355
• Copy Paste: [[2410.02355]] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models(https://arxiv.org/abs/2410.02355)
• Keywords: large language model
• Abstract: Large language models (LLMs) often exhibit hallucinations due to incorrect or outdated knowledge. Hence, model editing methods have emerged to enable targeted knowledge updates. To achieve this, a prevailing paradigm is the locating-then-editing approach, which first locates influential parameters and then edits them by introducing a perturbation. While effective, current studies have demonstrated that this perturbation inevitably disrupt the originally preserved knowledge within LLMs, especially in sequential editing scenarios. To address this, we introduce AlphaEdit, a novel solution that projects perturbation onto the null space of the preserved knowledge before applying it to the parameters. We theoretically prove that this projection ensures the output of post-edited LLMs remains unchanged when queried about the preserved knowledge, thereby mitigating the issue of disruption. Extensive experiments on various LLMs, including LLaMA3, GPT2-XL, and GPT-J, show that AlphaEdit boosts the performance of most locating-then-editing methods by an average of 36.4% with a single line of additional code for projection solely. Our code is available at: this https URL.

Title: A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond

• Authors: Shubhi Bansal, Sreeharish A, Madhava Prasath J, Manikandan S, Sreekanth Madisetty, Mohammad Zia Ur Rehman, Chandravardhan Singh Raghaw, Gaurav Duggal, Nagendra Kumar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02362
• Pdf URL: https://arxiv.org/pdf/2410.02362
• Copy Paste: [[2410.02362]] A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond(https://arxiv.org/abs/2410.02362)
• Keywords: transformer, segmentation
• Abstract: Mamba, a special case of the State Space Model, is gaining popularity as an alternative to template-based deep learning approaches in medical image analysis. While transformers are powerful architectures, they have drawbacks, including quadratic computational complexity and an inability to address long-range dependencies efficiently. This limitation affects the analysis of large and complex datasets in medical imaging, where there are many spatial and temporal relationships. In contrast, Mamba offers benefits that make it well-suited for medical image analysis. It has linear time complexity, which is a significant improvement over transformers. Mamba processes longer sequences without attention mechanisms, enabling faster inference and requiring less memory. Mamba also demonstrates strong performance in merging multimodal data, improving diagnosis accuracy and patient outcomes. The organization of this paper allows readers to appreciate the capabilities of Mamba in medical imaging step by step. We begin by defining core concepts of SSMs and models, including S4, S5, and S6, followed by an exploration of Mamba architectures such as pure Mamba, U-Net variants, and hybrid models with convolutional neural networks, transformers, and Graph Neural Networks. We also cover Mamba optimizations, techniques and adaptations, scanning, datasets, applications, experimental results, and conclude with its challenges and future directions in medical imaging. This review aims to demonstrate the transformative potential of Mamba in overcoming existing barriers within medical imaging while paving the way for innovative advancements in the field. A comprehensive list of Mamba architectures applied in the medical field, reviewed in this work, is available at Github.

Title: From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learning

• Authors: Haodong Xie, Rahul Singh Maharjan, Federico Tavella, Angelo Cangelosi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02365
• Pdf URL: https://arxiv.org/pdf/2410.02365
• Copy Paste: [[2410.02365]] From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learning(https://arxiv.org/abs/2410.02365)
• Keywords: generative
• Abstract: Understanding and manipulating concrete and abstract concepts is fundamental to human intelligence. Yet, they remain challenging for artificial agents. This paper introduces a multimodal generative approach to high order abstract concept learning, which integrates visual and categorical linguistic information from concrete ones. Our model initially grounds subordinate level concrete concepts, combines them to form basic level concepts, and finally abstracts to superordinate level concepts via the grounding of basic-level concepts. We evaluate the model language learning ability through language-to-visual and visual-to-language tests with high order abstract concepts. Experimental results demonstrate the proficiency of the model in both language understanding and language naming tasks.

Title: SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration

• Authors: Jintao Zhang, Jia wei, Pengle Zhang, Jun Zhu, Jianfei Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02367
• Pdf URL: https://arxiv.org/pdf/2410.02367
• Copy Paste: [[2410.02367]] SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration(https://arxiv.org/abs/2410.02367)
• Keywords: transformer
• Abstract: The transformer architecture predominates across various models. As the heart of the transformer, attention has a computational complexity of O(N^2), compared to O(N) for linear transformations. When handling large sequence lengths, attention becomes the primary time-consuming component. Although quantization has proven to be an effective method for accelerating model inference, existing quantization methods primarily focus on optimizing the linear layer. In response, we first analyze the feasibility of quantization in attention detailedly. Following that, we propose SageAttention, a highly efficient and accurate quantization method for attention. The OPS (operations per second) of our approach outperforms FlashAttention2 and xformers by about 2.1 times and 2.7 times, respectively. SageAttention also achieves superior accuracy performance over FlashAttention3. Comprehensive experiments confirm that our approach incurs almost no end-to-end metrics loss across diverse models, including those for large language processing, image generation, and video generation.

Title: Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation

• Authors: Muzhi Zhu, Yang Liu, Zekai Luo, Chenchen Jing, Hao Chen, Guangkai Xu, Xinlong Wang, Chunhua Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02369
• Pdf URL: https://arxiv.org/pdf/2410.02369
• Copy Paste: [[2410.02369]] Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation(https://arxiv.org/abs/2410.02369)
• Keywords: diffusion, generative, large language model, segmentation
• Abstract: The Diffusion Model has not only garnered noteworthy achievements in the realm of image generation but has also demonstrated its potential as an effective pretraining method utilizing unlabeled data. Drawing from the extensive potential unveiled by the Diffusion Model in both semantic correspondence and open vocabulary segmentation, our work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation. Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has evolved into In-context Segmentation tasks, morphing into a crucial element in assessing generalist segmentation models. In this context, we concentrate on Few-shot Semantic Segmentation, establishing a solid foundation for the future development of a Diffusion-based generalist model for segmentation. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework. Subsequently, we delve deeper into optimizing the infusion of information from the support mask and simultaneously re-evaluating how to provide reasonable supervision from the query mask. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior. Experimental results demonstrate that our method significantly outperforms the previous SOTA models in multiple settings.

Title: Unveiling AI's Blind Spots: An Oracle for In-Domain, Out-of-Domain, and Adversarial Errors

• Authors: Shuangpeng Han, Mengmi Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02384
• Pdf URL: https://arxiv.org/pdf/2410.02384
• Copy Paste: [[2410.02384]] Unveiling AI's Blind Spots: An Oracle for In-Domain, Out-of-Domain, and Adversarial Errors(https://arxiv.org/abs/2410.02384)
• Keywords: transformer
• Abstract: AI models make mistakes when recognizing images-whether in-domain, out-of-domain, or adversarial. Predicting these errors is critical for improving system reliability, reducing costly mistakes, and enabling proactive corrections in real-world applications such as healthcare, finance, and autonomous systems. However, understanding what mistakes AI models make, why they occur, and how to predict them remains an open challenge. Here, we conduct comprehensive empirical evaluations using a "mentor" model-a deep neural network designed to predict another model's errors. Our findings show that the mentor model excels at learning from a mentee's mistakes on adversarial images with small perturbations and generalizes effectively to predict in-domain and out-of-domain errors of the mentee. Additionally, transformer-based mentor models excel at predicting errors across various mentee architectures. Subsequently, we draw insights from these observations and develop an "oracle" mentor model, dubbed SuperMentor, that achieves 78% accuracy in predicting errors across different error types. Our error prediction framework paves the way for future research on anticipating and correcting AI model behaviours, ultimately increasing trust in AI systems. All code, models, and data will be made publicly available.

Title: Online Multi-Label Classification under Noisy and Changing Label Distribution

• Authors: Yizhang Zou, Xuegang Hu, Peipei Li, Jun Hu, You Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02394
• Pdf URL: https://arxiv.org/pdf/2410.02394
• Copy Paste: [[2410.02394]] Online Multi-Label Classification under Noisy and Changing Label Distribution(https://arxiv.org/abs/2410.02394)
• Keywords: robust
• Abstract: Multi-label data stream usually contains noisy labels in the real-world applications, namely occuring in both relevant and irrelevant labels. However, existing online multi-label classification methods are mostly limited in terms of label quality and fail to deal with the case of noisy labels. On the other hand, the ground-truth label distribution may vary with the time changing, which is hidden in the observed noisy label distribution and difficult to track, posing a major challenge for concept drift adaptation. Motivated by this, we propose an online multi-label classification algorithm under Noisy and Changing Label Distribution (NCLD). The convex objective is designed to simultaneously model the label scoring and the label ranking for high accuracy, whose robustness to NCLD benefits from three novel works: 1) The local feature graph is used to reconstruct the label scores jointly with the observed labels, and an unbiased ranking loss is derived and applied to learn reliable ranking information. 2) By detecting the difference between two adjacent chunks with the unbiased label cardinality, we identify the change in the ground-truth label distribution and reset the ranking or all information learned from the past to match the new distribution. 3) Efficient and accurate updating is achieved based on the updating rule derived from the closed-form optimal model solution. Finally, empirical experimental results validate the effectiveness of our method in classifying instances under NCLD.

Title: Parameter Competition Balancing for Model Merging

• Authors: Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, Min Zhang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02396
• Pdf URL: https://arxiv.org/pdf/2410.02396
• Copy Paste: [[2410.02396]] Parameter Competition Balancing for Model Merging(https://arxiv.org/abs/2410.02396)
• Keywords: large language model
• Abstract: While fine-tuning pretrained models has become common practice, these models often underperform outside their specific domains. Recently developed model merging techniques enable the direct integration of multiple models, each fine-tuned for distinct tasks, into a single model. This strategy promotes multitasking capabilities without requiring retraining on the original datasets. However, existing methods fall short in addressing potential conflicts and complex correlations between tasks, especially in parameter-level adjustments, posing a challenge in effectively balancing parameter competition across various tasks. This paper introduces an innovative technique named PCB-Merging (Parameter Competition Balancing), a lightweight and training-free technique that adjusts the coefficients of each parameter for effective model merging. PCB-Merging employs intra-balancing to gauge parameter significance within individual tasks and inter-balancing to assess parameter similarities across different tasks. Parameters with low importance scores are dropped, and the remaining ones are rescaled to form the final merged model. We assessed our approach in diverse merging scenarios, including cross-task, cross-domain, and cross-training configurations, as well as out-of-domain generalization. The experimental results reveal that our approach achieves substantial performance enhancements across multiple modalities, domains, model sizes, number of tasks, fine-tuning forms, and large language models, outperforming existing model merging methods. The code is publicly available at: \url{this https URL}.

Title: SynCo: Synthetic Hard Negatives in Contrastive Learning for Better Unsupervised Visual Representations

• Authors: Nikolaos Giakoumoglou, Tania Stathaki
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02401
• Pdf URL: https://arxiv.org/pdf/2410.02401
• Copy Paste: [[2410.02401]] SynCo: Synthetic Hard Negatives in Contrastive Learning for Better Unsupervised Visual Representations(https://arxiv.org/abs/2410.02401)
• Keywords: segmentation
• Abstract: Contrastive learning has become a dominant approach in self-supervised visual representation learning, with hard negatives-samples that closely resemble the anchor-being key to enhancing the discriminative power of learned representations. However, efficiently leveraging hard negatives remains a challenge due to the difficulty in identifying and incorporating them without significantly increasing computational costs. To address this, we introduce SynCo (Synthetic Negatives in Contrastive learning), a novel contrastive learning approach that improves model performance by generating synthetic hard negatives. Built on the MoCo framework, SynCo introduces six novel strategies for creating diverse synthetic hard negatives that can be generated on-the-fly with minimal computational overhead. SynCo achieves faster training and better representation learning, achieving a top-1 accuracy of 68.1% in ImageNet linear evaluation after only 200 epochs on pretraining, surpassing MoCo's 67.5% with the same ResNet-50 encoder. Additionally, it transfers more effectively to detection tasks: on the PASCAL VOC, it outperforms both the supervised baseline and MoCo, achieving an AP of 82.5%; on the COCO dataset, it sets a new benchmark with 40.4% AP for bounding box detection and 35.4% AP for instance segmentation. Our synthetic hard negative generation procedure significantly enhances the quality of visual representations learned through self-supervised contrastive learning. Code is available at this https URL.

Title: Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models

• Authors: Seyedmorteza Sadat, Otmar Hilliges, Romann M. Weber
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02416
• Pdf URL: https://arxiv.org/pdf/2410.02416
• Copy Paste: [[2410.02416]] Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models(https://arxiv.org/abs/2410.02416)
• Keywords: diffusion
• Abstract: Classifier-free guidance (CFG) is crucial for improving both generation quality and alignment between the input condition and final output in diffusion models. While a high guidance scale is generally required to enhance these aspects, it also causes oversaturation and unrealistic artifacts. In this paper, we revisit the CFG update rule and introduce modifications to address this issue. We first decompose the update term in CFG into parallel and orthogonal components with respect to the conditional model prediction and observe that the parallel component primarily causes oversaturation, while the orthogonal component enhances image quality. Accordingly, we propose down-weighting the parallel component to achieve high-quality generations without oversaturation. Additionally, we draw a connection between CFG and gradient ascent and introduce a new rescaling and momentum method for the CFG update rule based on this insight. Our approach, termed adaptive projected guidance (APG), retains the quality-boosting advantages of CFG while enabling the use of higher guidance scales without oversaturation. APG is easy to implement and introduces practically no additional computational overhead to the sampling process. Through extensive experiments, we demonstrate that APG is compatible with various conditional diffusion models and samplers, leading to improved FID, recall, and saturation scores while maintaining precision comparable to CFG, making our method a superior plug-and-play alternative to standard classifier-free guidance.

Title: MenakBERT -- Hebrew Diacriticizer

• Authors: Ido Cohen, Jacob Gidron, Idan Pinto
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02417
• Pdf URL: https://arxiv.org/pdf/2410.02417
• Copy Paste: [[2410.02417]] MenakBERT -- Hebrew Diacriticizer(https://arxiv.org/abs/2410.02417)
• Keywords: transformer
• Abstract: Diacritical marks in the Hebrew language give words their vocalized form. The task of adding diacritical marks to plain Hebrew text is still dominated by a system that relies heavily on human-curated resources. Recent models trained on diacritized Hebrew texts still present a gap in performance. We use a recently developed char-based PLM to narrowly bridge this gap. Presenting MenakBERT, a character level transformer pretrained on Hebrew text and fine-tuned to produce diacritical marks for Hebrew sentences. We continue to show how finetuning a model for diacritizing transfers to a task such as part of speech tagging.

Title: LoGDesc: Local geometric features aggregation for robust point cloud registration

• Authors: Karim Slimani, Brahim Tamadazte, Catherine Achard
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02420
• Pdf URL: https://arxiv.org/pdf/2410.02420
• Copy Paste: [[2410.02420]] LoGDesc: Local geometric features aggregation for robust point cloud registration(https://arxiv.org/abs/2410.02420)
• Keywords: robust
• Abstract: This paper introduces a new hybrid descriptor for 3D point matching and point cloud registration, combining local geometrical properties and learning-based feature propagation for each point's neighborhood structure description. The proposed architecture first extracts prior geometrical information by computing each point's planarity, anisotropy, and omnivariance using a Principal Components Analysis (PCA). This prior information is completed by a descriptor based on the normal vectors estimated thanks to constructing a neighborhood based on triangles. The final geometrical descriptor is propagated between the points using local graph convolutions and attention mechanisms. The new feature extractor is evaluated on ModelNet40, Bunny Stanford dataset, KITTI and MVP (Multi-View Partial)-RG for point cloud registration and shows interesting results, particularly on noisy and low overlapping point clouds.

Title: PnP-Flow: Plug-and-Play Image Restoration with Flow Matching

• Authors: Ségolène Martin, Anne Gagneux, Paul Hagemann, Gabriele Steidl
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02423
• Pdf URL: https://arxiv.org/pdf/2410.02423
• Copy Paste: [[2410.02423]] PnP-Flow: Plug-and-Play Image Restoration with Flow Matching(https://arxiv.org/abs/2410.02423)
• Keywords: generative
• Abstract: In this paper, we introduce Plug-and-Play (PnP) Flow Matching, an algorithm for solving imaging inverse problems. PnP methods leverage the strength of pre-trained denoisers, often deep neural networks, by integrating them in optimization schemes. While they achieve state-of-the-art performance on various inverse problems in imaging, PnP approaches face inherent limitations on more generative tasks like inpainting. On the other hand, generative models such as Flow Matching pushed the boundary in image sampling yet lack a clear method for efficient use in image restoration. We propose to combine the PnP framework with Flow Matching (FM) by defining a time-dependent denoiser using a pre-trained FM model. Our algorithm alternates between gradient descent steps on the data-fidelity term, reprojections onto the learned FM path, and denoising. Notably, our method is computationally efficient and memory-friendly, as it avoids backpropagation through ODEs and trace computations. We evaluate its performance on denoising, super-resolution, deblurring, and inpainting tasks, demonstrating superior results compared to existing PnP algorithms and Flow Matching based state-of-the-art methods.

Title: Learning the Latent Rules of a Game from Data: A Chess Story

• Authors: Ben Fauber
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02426
• Pdf URL: https://arxiv.org/pdf/2410.02426
• Copy Paste: [[2410.02426]] Learning the Latent Rules of a Game from Data: A Chess Story(https://arxiv.org/abs/2410.02426)
• Keywords: generative
• Abstract: We demonstrate that small pretrained foundational generative language models with millions of parameters can learn the latent rules of a process from data associated with the process. Inspired by Stefan Zweig's novella "Schachnovelle," also known as "The Royal Game" in English, we show that 28M and 125M parameter pretrained foundational small language models (SLMs) can be instruction fine-tuned with 1,000-to-1,000,000 examples to learn the rules of chess, propose legal moves, and accurately solve chess problems. We also explore the impact of successive language model fine-tuning epochs on improved outcomes and demonstrate reductions in model hallucinations by increasing the number of instruction fine-tuning examples.

Title: Collective Critics for Creative Story Generation

• Authors: Minwook Bae, Hyounghun Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02428
• Pdf URL: https://arxiv.org/pdf/2410.02428
• Copy Paste: [[2410.02428]] Collective Critics for Creative Story Generation(https://arxiv.org/abs/2410.02428)
• Keywords: large language model
• Abstract: Generating a long story of several thousand words with narrative coherence using Large Language Models (LLMs) has been a challenging task. Previous research has addressed this challenge by proposing different frameworks that create a story plan and generate a long story based on that plan. However, these frameworks have been mainly focusing on maintaining narrative coherence in stories, often overlooking creativity in story planning and the expressiveness of the stories generated from those plans, which are desirable properties to captivate readers' interest. In this paper, we propose Collective Critics for Creative Story Generation framework (CritiCS), which is composed of plan refining stage (CrPlan) and story generation stage (CrText), to integrate a collective revision mechanism that promotes those properties into long-form story generation process. Specifically, in each stage, a group of LLM critics and one leader collaborate to incrementally refine drafts of plan and story throughout multiple rounds. Extensive human evaluation shows that the CritiCS can significantly enhance story creativity and reader engagement, while also maintaining narrative coherence. Furthermore, the design of the framework allows active participation from human writers in any role within the critique process, enabling interactive human-machine collaboration in story writing.

Title: Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization

• Authors: Mingyang Wang, Lukas Lange, Heike Adel, Jannik Strötgen, Hinrich Schütze
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02433
• Pdf URL: https://arxiv.org/pdf/2410.02433
• Copy Paste: [[2410.02433]] Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization(https://arxiv.org/abs/2410.02433)
• Keywords: large language model
• Abstract: To ensure large language models contain up-to-date knowledge, they need to be updated regularly. However, model editing is challenging as it might also affect knowledge that is unrelated to the new data. State-of-the-art methods identify parameters associated with specific knowledge and then modify them via direct weight updates. However, these locate-and-edit methods suffer from heavy computational overhead and lack theoretical validation. In contrast, directly fine-tuning the model on requested edits affects the model's behavior on unrelated knowledge, and significantly damages the model's generation fluency and consistency. To address these challenges, we propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization. Evaluations on three model editing benchmarks show that SAUL is a practical and reliable solution for model editing outperforming state-of-the-art methods while maintaining generation quality and reducing computational overhead.

Title: Optimizing Adaptive Attacks against Content Watermarks for Language Models

• Authors: Abdulrahman Diaa, Toluwani Aremu, Nils Lukas
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02440
• Pdf URL: https://arxiv.org/pdf/2410.02440
• Copy Paste: [[2410.02440]] Optimizing Adaptive Attacks against Content Watermarks for Language Models(https://arxiv.org/abs/2410.02440)
• Keywords: security, attack, robust, watermark, large language model
• Abstract: Large Language Models (LLMs) can be \emph{misused} to spread online spam and misinformation. Content watermarking deters misuse by hiding a message in model-generated outputs, enabling their detection using a secret watermarking key. Robustness is a core security property, stating that evading detection requires (significant) degradation of the content's quality. Many LLM watermarking methods have been proposed, but robustness is tested only against \emph{non-adaptive} attackers who lack knowledge of the watermarking method and can find only suboptimal attacks. We formulate the robustness of LLM watermarking as an objective function and propose preference-based optimization to tune \emph{adaptive} attacks against the specific watermarking method. Our evaluation shows that (i) adaptive attacks substantially outperform non-adaptive baselines. (ii) Even in a non-adaptive setting, adaptive attacks optimized against a few known watermarks remain highly effective when tested against other unseen watermarks, and (iii) optimization-based attacks are practical and require less than seven GPU hours. Our findings underscore the need to test robustness against adaptive attackers.

Title: Towards a Self-rescuing System for UAVs Under GNSS Attack

• Authors: Giulio Rigoni, Nicola Scremin, Mauro Conti
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02442
• Pdf URL: https://arxiv.org/pdf/2410.02442
• Copy Paste: [[2410.02442]] Towards a Self-rescuing System for UAVs Under GNSS Attack(https://arxiv.org/abs/2410.02442)
• Keywords: security, attack
• Abstract: There has been substantial growth in the UAV market along with an expansion in their applications. However, the successful execution of a UAV mission is very often dependent on the use of a GNSS. Unfortunately, the vulnerability of GNSS signals, due to their lack of encryption and authentication, poses a significant cybersecurity issue. This vulnerability makes various attacks, particularly the "GNSS spoofing attack," and "GNSS jamming attack" easily executable. Generally speaking, during this attack, the drone is manipulated into altering its path, usually resulting in an immediate forced landing or crash. As far as we know, we are the first to propose a lightweight-solution that enable a drone to autonomously rescue itself, assuming it is under GNSS attack and the GNSS is no longer available, and return safely to its initial takeoff position, thereby preventing any potential crashes. During the flight, wind plays a critical role as it can instantaneously alter the drone's position. To solve this problem, we have devised a highly effective 2-phases solution: (i) Forward Phase, for monitoring and recording the forward journey, and (ii) Backward Phase, that generates a backward route, based on the Forward Phase and wind presence. The final solution ensures strong performance in consistently returning the drone to the original position, even in wind situations, while maintaining a very fast computation time.

Title: Clinnova Federated Learning Proof of Concept: Key Takeaways from a Cross-border Collaboration

• Authors: Julia Alekseenko, Bram Stieltjes, Michael Bach, Melanie Boerries, Oliver Opitz, Alexandros Karargyris, Nicolas Padoy
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02443
• Pdf URL: https://arxiv.org/pdf/2410.02443
• Copy Paste: [[2410.02443]] Clinnova Federated Learning Proof of Concept: Key Takeaways from a Cross-border Collaboration(https://arxiv.org/abs/2410.02443)
• Keywords: federate, segmentation
• Abstract: Clinnova, a collaborative initiative involving France, Germany, Switzerland, and Luxembourg, is dedicated to unlocking the power of precision medicine through data federation, standardization, and interoperability. This European Greater Region initiative seeks to create an interoperable European standard using artificial intelligence (AI) and data science to enhance healthcare outcomes and efficiency. Key components include multidisciplinary research centers, a federated biobanking strategy, a digital health innovation platform, and a federated AI strategy. It targets inflammatory bowel disease, rheumatoid diseases, and multiple sclerosis (MS), emphasizing data quality to develop AI algorithms for personalized treatment and translational research. The IHU Strasbourg (Institute of Minimal-invasive Surgery) has the lead in this initiative to develop the federated learning (FL) proof of concept (POC) that will serve as a foundation for advancing AI in healthcare. At its core, Clinnova-MS aims to enhance MS patient care by using FL to develop more accurate models that detect disease progression, guide interventions, and validate digital biomarkers across multiple sites. This technical report presents insights and key takeaways from the first cross-border federated POC on MS segmentation of MRI images within the Clinnova framework. While our work marks a significant milestone in advancing MS segmentation through cross-border collaboration, it also underscores the importance of addressing technical, logistical, and ethical considerations to realize the full potential of FL in healthcare settings.

Title: Personalized Federated Learning for Generative AI-Assisted Semantic Communications

• Authors: Yubo Peng, Feibo Jiang, Li Dong, Kezhi Wang, Kun Yang
• Subjects: cs.LG, cs.DC, cs.IT
• Abstract URL: https://arxiv.org/abs/2410.02450
• Pdf URL: https://arxiv.org/pdf/2410.02450
• Copy Paste: [[2410.02450]] Personalized Federated Learning for Generative AI-Assisted Semantic Communications(https://arxiv.org/abs/2410.02450)
• Keywords: privacy, federate, generative
• Abstract: Semantic Communication (SC) focuses on transmitting only the semantic information rather than the raw data. This approach offers an efficient solution to the issue of spectrum resource utilization caused by the various intelligent applications on Mobile Users (MUs). Generative Artificial Intelligence (GAI) models have recently exhibited remarkable content generation and signal processing capabilities, presenting new opportunities for enhancing SC. Therefore, we propose a GAI-assisted SC (GSC) model deployed between MUs and the Base Station (BS). Then, to train the GSC model using the local data of MUs while ensuring privacy and accommodating heterogeneous requirements of MUs, we introduce Personalized Semantic Federated Learning (PSFL). This approach incorporates a novel Personalized Local Distillation (PLD) and Adaptive Global Pruning (AGP). In PLD, each MU selects a personalized GSC model as a mentor tailored to its local resources and a unified Convolutional Neural Networks (CNN)-based SC (CSC) model as a student. This mentor model is then distilled into the student model for global aggregation. In AGP, we perform network pruning on the aggregated global model according to real-time communication environments, reducing communication energy. Finally, numerical results demonstrate the feasibility and efficiency of the proposed PSFL scheme.

Title: Recurrent Few-Shot model for Document Verification

• Authors: Maxime Talarmain, Carlos Boned, Sanket Biswas, Oriol Ramos
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02456
• Pdf URL: https://arxiv.org/pdf/2410.02456
• Copy Paste: [[2410.02456]] Recurrent Few-Shot model for Document Verification(https://arxiv.org/abs/2410.02456)
• Keywords: robust
• Abstract: General-purpose ID, or travel, document image- and video-based verification systems have yet to achieve good enough performance to be considered a solved problem. There are several factors that negatively impact their performance, including low-resolution images and videos and a lack of sufficient data to train the models. This task is particularly challenging when dealing with unseen class of ID, or travel, documents. In this paper we address this task by proposing a recurrent-based model able to detect forged documents in a few-shot scenario. The recurrent architecture makes the model robust to document resolution variability. Moreover, the few-shot approach allow the model to perform well even for unseen class of documents. Preliminary results on the SIDTD and Findit datasets show good performance of this model for this task.

Title: Scalable Differential Privacy Mechanisms for Real-Time Machine Learning Applications

• Authors: Jessica Smith, David Williams, Emily Brown
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02462
• Pdf URL: https://arxiv.org/pdf/2410.02462
• Copy Paste: [[2410.02462]] Scalable Differential Privacy Mechanisms for Real-Time Machine Learning Applications(https://arxiv.org/abs/2410.02462)
• Keywords: privacy, protect, robust, large language model
• Abstract: Large language models (LLMs) are increasingly integrated into real-time machine learning applications, where safeguarding user privacy is paramount. Traditional differential privacy mechanisms often struggle to balance privacy and accuracy, particularly in fast-changing environments with continuously flowing data. To address these issues, we introduce Scalable Differential Privacy (SDP), a framework tailored for real-time machine learning that emphasizes both robust privacy guarantees and enhanced model performance. SDP employs a hierarchical architecture to facilitate efficient noise aggregation across various learning agents. By integrating adaptive noise scheduling and gradient compression methods, our approach minimizes performance degradation while ensuring significant privacy protection. Extensive experiments on diverse datasets reveal that SDP maintains high accuracy levels while applying differential privacy effectively, showcasing its suitability for deployment in sensitive domains. This advancement points towards the potential for widespread adoption of privacy-preserving techniques in machine learning workflows.

Title: Response Tuning: Aligning Large Language Models without Instruction

• Authors: Seokhyun An, Hyounghun Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02465
• Pdf URL: https://arxiv.org/pdf/2410.02465
• Copy Paste: [[2410.02465]] Response Tuning: Aligning Large Language Models without Instruction(https://arxiv.org/abs/2410.02465)
• Keywords: large language model
• Abstract: Instruction tuning-supervised fine-tuning using instruction-response pairs-is a foundational step in transitioning pre-trained Large Language Models (LLMs) into helpful and safe chat assistants. Our hypothesis is that establishing an adequate output space can enable such a transition given the capabilities inherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT), which eliminates the instruction-conditioning step in instruction tuning and solely focuses on response space supervision. Our experiments demonstrate that RT models, trained only using responses, can effectively respond to a wide range of instructions and exhibit helpfulness comparable to that of their instruction-tuned counterparts. Furthermore, we observe that controlling the training response distribution can significantly improve their user preference or elicit target behaviors such as refusing assistance for unsafe queries. Our findings illuminate the role of establishing an adequate output space in alignment, highlighting the potential of the extensive inherent capabilities of pre-trained LLMs.

Title: Towards a Theoretical Understanding of Memorization in Diffusion Models

• Authors: Yunhao Chen, Xingjun Ma, Difan Zou, Yu-Gang Jiang
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02467
• Pdf URL: https://arxiv.org/pdf/2410.02467
• Copy Paste: [[2410.02467]] Towards a Theoretical Understanding of Memorization in Diffusion Models(https://arxiv.org/abs/2410.02467)
• Keywords: extraction, diffusion, generative
• Abstract: As diffusion probabilistic models (DPMs) are being employed as mainstream models for Generative Artificial Intelligence (GenAI), the study of their memorization of training data has attracted growing attention. Existing works in this direction aim to establish an understanding of whether or to what extent DPMs learn via memorization. Such an understanding is crucial for identifying potential risks of data leakage and copyright infringement in diffusion models and, more importantly, for trustworthy application of GenAI. Existing works revealed that conditional DPMs are more prone to training data memorization than unconditional DPMs, and the motivated data extraction methods are mostly for conditional DPMs. However, these understandings are primarily empirical, and extracting training data from unconditional models has been found to be extremely challenging. In this work, we provide a theoretical understanding of memorization in both conditional and unconditional DPMs under the assumption of model convergence. Our theoretical analysis indicates that extracting data from unconditional models can also be effective by constructing a proper surrogate condition. Based on this result, we propose a novel data extraction method named \textbf{Surrogate condItional Data Extraction (SIDE)} that leverages a time-dependent classifier trained on the generated data as a surrogate condition to extract training data from unconditional DPMs. Empirical results demonstrate that our SIDE can extract training data in challenging scenarios where previous methods fail, and it is, on average, over 50\% more effective across different scales of the CelebA dataset.

Title: Meta-Models: An Architecture for Decoding LLM Behaviors Through Interpreted Embeddings and Natural Language

• Authors: Anthony Costarelli, Mat Allen, Severin Field, Joshua Clymer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02472
• Pdf URL: https://arxiv.org/pdf/2410.02472
• Copy Paste: [[2410.02472]] Meta-Models: An Architecture for Decoding LLM Behaviors Through Interpreted Embeddings and Natural Language(https://arxiv.org/abs/2410.02472)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) become increasingly integrated into our daily lives, the potential harms from deceptive behavior underlie the need for faithfully interpreting their decision-making. While traditional probing methods have shown some effectiveness, they remain best for narrowly scoped tasks while more comprehensive explanations are still necessary. To this end, we investigate meta-models-an architecture using a "meta-model" that takes activations from an "input-model" and answers natural language questions about the input-model's behaviors. We evaluate the meta-model's ability to generalize by training them on selected task types and assessing their out-of-distribution performance in deceptive scenarios. Our findings show that meta-models generalize well to out-of-distribution tasks and point towards opportunities for future research in this area.

Title: Event-Customized Image Generation

• Authors: Zhen Wang, Yilei Jiang, Dong Zheng, Jun Xiao, Long Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02483
• Pdf URL: https://arxiv.org/pdf/2410.02483
• Copy Paste: [[2410.02483]] Event-Customized Image Generation(https://arxiv.org/abs/2410.02483)
• Keywords: diffusion
• Abstract: Customized Image Generation, generating customized images with user-specified concepts, has raised significant attention due to its creativity and novelty. With impressive progress achieved in subject customization, some pioneer works further explored the customization of action and interaction beyond entity (i.e., human, animal, and object) appearance. However, these approaches only focus on basic actions and interactions between two entities, and their effects are limited by insufficient ''exactly same'' reference images. To extend customized image generation to more complex scenes for general real-world applications, we propose a new task: event-customized image generation. Given a single reference image, we define the ''event'' as all specific actions, poses, relations, or interactions between different entities in the scene. This task aims at accurately capturing the complex event and generating customized images with various target entities. To solve this task, we proposed a novel training-free event customization method: FreeEvent. Specifically, FreeEvent introduces two extra paths alongside the general diffusion denoising process: 1) Entity switching path: it applies cross-attention guidance and regulation for target entity generation. 2) Event transferring path: it injects the spatial feature and self-attention maps from the reference image to the target image for event generation. To further facilitate this new task, we collected two evaluation benchmarks: SWiG-Event and Real-Event. Extensive experiments and ablations have demonstrated the effectiveness of FreeEvent.

Title: Encryption-Friendly LLM Architecture

• Authors: Donghwan Rho, Taeseong Kim, Minje Park, Jung Woo Kim, Hyunsik Chae, Jung Hee Cheon, Ernest K. Ryu
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02486
• Pdf URL: https://arxiv.org/pdf/2410.02486
• Copy Paste: [[2410.02486]] Encryption-Friendly LLM Architecture(https://arxiv.org/abs/2410.02486)
• Keywords: privacy, protect, transformer, large language model
• Abstract: Large language models (LLMs) offer personalized responses based on user interactions, but this use case raises serious privacy concerns. Homomorphic encryption (HE) is a cryptographic protocol supporting arithmetic computations in encrypted states and provides a potential solution for privacy-preserving machine learning (PPML). However, the computational intensity of transformers poses challenges for applying HE to LLMs. In this work, we propose a modified HE-friendly transformer architecture with an emphasis on inference following personalized (private) fine-tuning. Utilizing LoRA fine-tuning and Gaussian kernels, we achieve significant computational speedups -- 6.94x for fine-tuning and 2.3x for inference -- while maintaining performance comparable to plaintext models. Our findings provide a viable proof of concept for offering privacy-preserving LLM services in areas where data protection is crucial.

Title: DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM

• Authors: Xuchen Li, Shiyu Hu, Xiaokun Feng, Dailing Zhang, Meiqi Wu, Jing Zhang, Kaiqi Huang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02492
• Pdf URL: https://arxiv.org/pdf/2410.02492
• Copy Paste: [[2410.02492]] DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM(https://arxiv.org/abs/2410.02492)
• Keywords: large language model
• Abstract: Visual language tracking (VLT) has emerged as a cutting-edge research area, harnessing linguistic data to enhance algorithms with multi-modal inputs and broadening the scope of traditional single object tracking (SOT) to encompass video understanding applications. Despite this, most VLT benchmarks still depend on succinct, human-annotated text descriptions for each video. These descriptions often fall short in capturing the nuances of video content dynamics and lack stylistic variety in language, constrained by their uniform level of detail and a fixed annotation frequency. As a result, algorithms tend to default to a "memorize the answer" strategy, diverging from the core objective of achieving a deeper understanding of video content. Fortunately, the emergence of large language models (LLMs) has enabled the generation of diverse text. This work utilizes LLMs to generate varied semantic annotations (in terms of text lengths and granularities) for representative SOT benchmarks, thereby establishing a novel multi-modal benchmark. Specifically, we (1) propose a new visual language tracking benchmark with diverse texts, named DTVLT, based on five prominent VLT and SOT benchmarks, including three sub-tasks: short-term tracking, long-term tracking, and global instance tracking. (2) We offer four granularity texts in our benchmark, considering the extent and density of semantic information. We expect this multi-granular generation strategy to foster a favorable environment for VLT and video understanding research. (3) We conduct comprehensive experimental analyses on DTVLT, evaluating the impact of diverse text on tracking performance and hope the identified performance bottlenecks of existing algorithms can support further research in VLT and video understanding. The proposed benchmark, experimental results and toolkit will be released gradually on this http URL.

Title: Dynamic Gradient Alignment for Online Data Mixing

• Authors: Simin Fan, David Grangier, Pierre Ablin
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02498
• Pdf URL: https://arxiv.org/pdf/2410.02498
• Copy Paste: [[2410.02498]] Dynamic Gradient Alignment for Online Data Mixing(https://arxiv.org/abs/2410.02498)
• Keywords: large language model
• Abstract: The composition of training data mixtures is critical for effectively training large language models (LLMs), as it directly impacts their performance on downstream tasks. Our goal is to identify an optimal data mixture to specialize an LLM for a specific task with access to only a few examples. Traditional approaches to this problem include ad-hoc reweighting methods, importance sampling, and gradient alignment techniques. This paper focuses on gradient alignment and introduces Dynamic Gradient Alignment (DGA), a scalable online gradient alignment algorithm. DGA dynamically estimates the pre-training data mixture on which the models' gradients align as well as possible with those of the model on the specific task. DGA is the first gradient alignment approach that incurs minimal overhead compared to standard pre-training and outputs a competitive model, eliminating the need for retraining the model. Experimentally, we demonstrate significant improvements over importance sampling in two key scenarios: (i) when the pre-training set is small and importance sampling overfits due to limited data; and (ii) when there is insufficient specialized data, trapping importance sampling on narrow pockets of data. Our findings underscore the effectiveness of gradient alignment methods in optimizing training data mixtures, particularly in data-constrained environments, and offer a practical solution for enhancing LLM performance on specific tasks with limited data availability.

Title: Defining Knowledge: Bridging Epistemology and Large Language Models

• Authors: Constanza Fierro, Ruchira Dhar, Filippos Stamatiou, Nicolas Garneau, Anders Søgaard
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02499
• Pdf URL: https://arxiv.org/pdf/2410.02499
• Copy Paste: [[2410.02499]] Defining Knowledge: Bridging Epistemology and Large Language Models(https://arxiv.org/abs/2410.02499)
• Keywords: large language model
• Abstract: Knowledge claims are abundant in the literature on large language models (LLMs); but can we say that GPT-4 truly "knows" the Earth is round? To address this question, we review standard definitions of knowledge in epistemology and we formalize interpretations applicable to LLMs. In doing so, we identify inconsistencies and gaps in how current NLP research conceptualizes knowledge with respect to epistemological frameworks. Additionally, we conduct a survey of 100 professional philosophers and computer scientists to compare their preferences in knowledge definitions and their views on whether LLMs can really be said to know. Finally, we suggest evaluation protocols for testing knowledge in accordance to the most relevant definitions.

Title: Dog-IQA: Standard-guided Zero-shot MLLM for Mix-grained Image Quality Assessment

• Authors: Kai Liu, Ziqing Zhang, Wenbo Li, Renjing Pei, Fenglong Song, Xiaohong Liu, Linghe Kong, Yulun Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02505
• Pdf URL: https://arxiv.org/pdf/2410.02505
• Copy Paste: [[2410.02505]] Dog-IQA: Standard-guided Zero-shot MLLM for Mix-grained Image Quality Assessment(https://arxiv.org/abs/2410.02505)
• Keywords: large language model
• Abstract: Image quality assessment (IQA) serves as the golden standard for all models' performance in nearly all computer vision fields. However, it still suffers from poor out-of-distribution generalization ability and expensive training costs. To address these problems, we propose Dog-IQA, a standard-guided zero-shot mix-grained IQA method, which is training-free and utilizes the exceptional prior knowledge of multimodal large language models (MLLMs). To obtain accurate IQA scores, namely scores consistent with humans, we design an MLLM-based inference pipeline that imitates human experts. In detail, Dog-IQA applies two techniques. First, Dog-IQA objectively scores with specific standards that utilize MLLM's behavior pattern and minimize the influence of subjective factors. Second, Dog-IQA comprehensively takes local semantic objects and the whole image as input and aggregates their scores, leveraging local and global information. Our proposed Dog-IQA achieves state-of-the-art (SOTA) performance compared with training-free methods, and competitive performance compared with training-based methods in cross-dataset scenarios. Our code and models will be available at this https URL.

Title: SAFLEX: Self-Adaptive Augmentation via Feature Label Extrapolation

• Authors: Mucong Ding, Bang An, Yuancheng Xu, Anirudh Satheesh, Furong Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02512
• Pdf URL: https://arxiv.org/pdf/2410.02512
• Copy Paste: [[2410.02512]] SAFLEX: Self-Adaptive Augmentation via Feature Label Extrapolation(https://arxiv.org/abs/2410.02512)
• Keywords: diffusion, generative
• Abstract: Data augmentation, a cornerstone technique in deep learning, is crucial in enhancing model performance, especially with scarce labeled data. While traditional techniques are effective, their reliance on hand-crafted methods limits their applicability across diverse data types and tasks. Although modern learnable augmentation methods offer increased adaptability, they are computationally expensive and challenging to incorporate within prevalent augmentation workflows. In this work, we present a novel, efficient method for data augmentation, effectively bridging the gap between existing augmentation strategies and emerging datasets and learning tasks. We introduce SAFLEX (Self-Adaptive Augmentation via Feature Label EXtrapolation), which learns the sample weights and soft labels of augmented samples provided by any given upstream augmentation pipeline, using a specifically designed efficient bilevel optimization algorithm. Remarkably, SAFLEX effectively reduces the noise and label errors of the upstream augmentation pipeline with a marginal computational cost. As a versatile module, SAFLEX excels across diverse datasets, including natural and medical images and tabular data, showcasing its prowess in few-shot learning and out-of-distribution generalization. SAFLEX seamlessly integrates with common augmentation strategies like RandAug, CutMix, and those from large pre-trained generative models like stable diffusion and is also compatible with frameworks such as CLIP's fine-tuning. Our findings highlight the potential to adapt existing augmentation pipelines for new data types and tasks, signaling a move towards more adaptable and resilient training frameworks.

Title: Minimax Group Fairness in Strategic Classification

• Authors: Emily Diana, Saeed Sharifi-Malvajerdi, Ali Vakilian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02513
• Pdf URL: https://arxiv.org/pdf/2410.02513
• Copy Paste: [[2410.02513]] Minimax Group Fairness in Strategic Classification(https://arxiv.org/abs/2410.02513)
• Keywords: robust, fair
• Abstract: In strategic classification, agents manipulate their features, at a cost, to receive a positive classification outcome from the learner's classifier. The goal of the learner in such settings is to learn a classifier that is robust to strategic manipulations. While the majority of works in this domain consider accuracy as the primary objective of the learner, in this work, we consider learning objectives that have group fairness guarantees in addition to accuracy guarantees. We work with the minimax group fairness notion that asks for minimizing the maximal group error rate across population groups. We formalize a fairness-aware Stackelberg game between a population of agents consisting of several groups, with each group having its own cost function, and a learner in the agnostic PAC setting in which the learner is working with a hypothesis class H. When the cost functions of the agents are separable, we show the existence of an efficient algorithm that finds an approximately optimal deterministic classifier for the learner when the number of groups is small. This algorithm remains efficient, both statistically and computationally, even when H is the set of all classifiers. We then consider cost functions that are not necessarily separable and show the existence of oracle-efficient algorithms that find approximately optimal randomized classifiers for the learner when H has finite strategic VC dimension. These algorithms work under the assumption that the learner is fully transparent: the learner draws a classifier from its distribution (randomized classifier) before the agents respond by manipulating their feature vectors. We highlight the effectiveness of such transparency in developing oracle-efficient algorithms. We conclude with verifying the efficacy of our algorithms on real data by conducting an experimental analysis.

Title: Semantic-Guided RL for Interpretable Feature Engineering

• Authors: Mohamed Bouadi, Arta Alavi, Salima Benbernou, Mourad Ouziri
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02519
• Pdf URL: https://arxiv.org/pdf/2410.02519
• Copy Paste: [[2410.02519]] Semantic-Guided RL for Interpretable Feature Engineering(https://arxiv.org/abs/2410.02519)
• Keywords: interpretability
• Abstract: The quality of Machine Learning (ML) models strongly depends on the input data, as such generating high-quality features is often required to improve the predictive accuracy. This process is referred to as Feature Engineering (FE). However, since manual feature engineering is time-consuming and requires case-by-case domain knowledge, Automated Feature Engineering (AutoFE) is crucial. A major challenge that remains is to generate interpretable features. To tackle this problem, we introduce SMART, a hybrid approach that uses semantic technologies to guide the generation of interpretable features through a two-step process: Exploitation and Exploration. The former uses Description Logics (DL) to reason on the semantics embedded in Knowledge Graphs (KG) to infer domain-specific features, while the latter exploits the knowledge graph to conduct a guided exploration of the search space through Deep Reinforcement Learning (DRL). Our experiments on public datasets demonstrate that SMART significantly improves prediction accuracy while ensuring a high level of interpretability.

Title: HiFiSeg: High-Frequency Information Enhanced Polyp Segmentation with Global-Local Vision Transformer

• Authors: Jingjing Ren, Xiaoyong Zhang, Lina Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02528
• Pdf URL: https://arxiv.org/pdf/2410.02528
• Copy Paste: [[2410.02528]] HiFiSeg: High-Frequency Information Enhanced Polyp Segmentation with Global-Local Vision Transformer(https://arxiv.org/abs/2410.02528)
• Keywords: transformer, segmentation
• Abstract: Numerous studies have demonstrated the strong performance of Vision Transformer (ViT)-based methods across various computer vision tasks. However, ViT models often struggle to effectively capture high-frequency components in images, which are crucial for detecting small targets and preserving edge details, especially in complex scenarios. This limitation is particularly challenging in colon polyp segmentation, where polyps exhibit significant variability in structure, texture, and shape. High-frequency information, such as boundary details, is essential for achieving precise semantic segmentation in this context. To address these challenges, we propose HiFiSeg, a novel network for colon polyp segmentation that enhances high-frequency information processing through a global-local vision transformer framework. HiFiSeg leverages the pyramid vision transformer (PVT) as its encoder and introduces two key modules: the global-local interaction module (GLIM) and the selective aggregation module (SAM). GLIM employs a parallel structure to fuse global and local information at multiple scales, effectively capturing fine-grained features. SAM selectively integrates boundary details from low-level features with semantic information from high-level features, significantly improving the model's ability to accurately detect and segment polyps. Extensive experiments on five widely recognized benchmark datasets demonstrate the effectiveness of HiFiSeg for polyp segmentation. Notably, the mDice scores on the challenging CVC-ColonDB and ETIS datasets reached 0.826 and 0.822, respectively, underscoring the superior performance of HiFiSeg in handling the specific complexities of this task.

Title: An Edge-Computing based Industrial Gateway for Industry 4.0 using ARM TrustZone Technology

• Authors: Sandeep Gupta
• Subjects: cs.CR, cs.DC, cs.ET
• Abstract URL: https://arxiv.org/abs/2410.02529
• Pdf URL: https://arxiv.org/pdf/2410.02529
• Copy Paste: [[2410.02529]] An Edge-Computing based Industrial Gateway for Industry 4.0 using ARM TrustZone Technology(https://arxiv.org/abs/2410.02529)
• Keywords: secure, security
• Abstract: Secure and efficient communication to establish a seamless nexus between the five levels of a typical automation pyramid is paramount to Industry 4.0. Specifically, vertical and horizontal integration of these levels is an overarching requirement to accelerate productivity and improve operational activities. Vertical integration can improve visibility, flexibility, and productivity by connecting systems and applications. Horizontal integration can provide better collaboration and adaptability by connecting internal production facilities, multi-site operations, and third-party partners in a supply chain. In this paper, we propose an Edge-computing-based Industrial Gateway for interfacing information technology and operational technology that can enable Industry 4.0 vertical and horizontal integration. Subsequently, we design and develop a working prototype to demonstrate a remote production-line maintenance use case with a strong focus on security aspects and the edge paradigm to bring computational resources and data storage closer to data sources.

Title: Exploiting HDMI and USB Ports for GPU Side-Channel Insights

• Authors: Sayed Erfan Arefin, Abdul Serwadda
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02539
• Pdf URL: https://arxiv.org/pdf/2410.02539
• Copy Paste: [[2410.02539]] Exploiting HDMI and USB Ports for GPU Side-Channel Insights(https://arxiv.org/abs/2410.02539)
• Keywords: security, defense, attack
• Abstract: Modern computers rely on USB and HDMI ports for connecting external peripherals and display devices. Despite their built-in security measures, these ports remain susceptible to passive power-based side-channel attacks. This paper presents a new class of attacks that exploit power consumption patterns at these ports to infer GPU activities. We develop a custom device that plugs into these ports and demonstrate that its high-resolution power measurements can drive successful inferences about GPU processes, such as neural network computations and video rendering. The ubiquitous presence of USB and HDMI ports allows for discreet placement of the device, and its non-interference with data channels ensures that no security alerts are triggered. Our findings underscore the need to reevaluate and strengthen the current generation of HDMI and USB port security defenses.

Title: Fair Decentralized Learning

• Authors: Sayan Biswas, Anne-Marie Kermarrec, Rishi Sharma, Thibaud Trinca, Martijn de Vos
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.02541
• Pdf URL: https://arxiv.org/pdf/2410.02541
• Copy Paste: [[2410.02541]] Fair Decentralized Learning(https://arxiv.org/abs/2410.02541)
• Keywords: fair
• Abstract: Decentralized learning (DL) is an emerging approach that enables nodes to collaboratively train a machine learning model without sharing raw data. In many application domains, such as healthcare, this approach faces challenges due to the high level of heterogeneity in the training data's feature space. Such feature heterogeneity lowers model utility and negatively impacts fairness, particularly for nodes with under-represented training data. In this paper, we introduce \textsc{Facade}, a clustering-based DL algorithm specifically designed for fair model training when the training data exhibits several distinct features. The challenge of \textsc{Facade} is to assign nodes to clusters, one for each feature, based on the similarity in the features of their local data, without requiring individual nodes to know apriori which cluster they belong to. \textsc{Facade} (1) dynamically assigns nodes to their appropriate clusters over time, and (2) enables nodes to collaboratively train a specialized model for each cluster in a fully decentralized manner. We theoretically prove the convergence of \textsc{Facade}, implement our algorithm, and compare it against three state-of-the-art baselines. Our experimental results on three datasets demonstrate the superiority of our approach in terms of model accuracy and fairness compared to all three competitors. Compared to the best-performing baseline, \textsc{Facade} on the CIFAR-10 dataset also reduces communication costs by 32.3\% to reach a target accuracy when cluster sizes are imbalanced.

Title: ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaboration

• Authors: Zixiang Wang, Yinghao Zhu, Huiya Zhao, Xiaochen Zheng, Tianlong Wang, Wen Tang, Yasha Wang, Chengwei Pan, Ewen M. Harrison, Junyi Gao, Liantao Ma
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02551
• Pdf URL: https://arxiv.org/pdf/2410.02551
• Copy Paste: [[2410.02551]] ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaboration(https://arxiv.org/abs/2410.02551)
• Keywords: large language model
• Abstract: We introduce ColaCare, a framework that enhances Electronic Health Record (EHR) modeling through multi-agent collaboration driven by Large Language Models (LLMs). Our approach seamlessly integrates domain-specific expert models with LLMs to bridge the gap between structured EHR data and text-based reasoning. Inspired by clinical consultations, ColaCare employs two types of agents: DoctorAgent and MetaAgent, which collaboratively analyze patient data. Expert models process and generate predictions from numerical EHR data, while LLM agents produce reasoning references and decision-making reports within the collaborative consultation framework. We additionally incorporate the Merck Manual of Diagnosis and Therapy (MSD) medical guideline within a retrieval-augmented generation (RAG) module for authoritative evidence support. Extensive experiments conducted on four distinct EHR datasets demonstrate ColaCare's superior performance in mortality prediction tasks, underscoring its potential to revolutionize clinical decision support systems and advance personalized precision medicine. The code, complete prompt templates, more case studies, etc. are publicly available at the anonymous link: this https URL.

Title: Assessing the Viability of Synthetic Physical Copy Detection Patterns on Different Imaging Systems

• Authors: Roman Chaban, Brian Pulfer, Slava Voloshynovskiy
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2410.02575
• Pdf URL: https://arxiv.org/pdf/2410.02575
• Copy Paste: [[2410.02575]] Assessing the Viability of Synthetic Physical Copy Detection Patterns on Different Imaging Systems(https://arxiv.org/abs/2410.02575)
• Keywords: security, robust
• Abstract: This paper explores the potential of synthetic physical Copy Detection Patterns (CDP) to improve the robustness of anti-counterfeiting systems. By leveraging synthetic physical CDP, we aim at enhancing security and cost-effectiveness across various real-world applications. Our research demonstrates that synthetic CDP offer substantial improvements in authentication accuracy compared to one based on traditional digital templates. We conducted extensive tests using both a scanner and a diverse range of mobile phones, validating our approach through ROC analysis. The results indicate that synthetic CDP can reliably differentiate between original and fake samples, making this approach a viable solution for real-world applications, though requires an additional research to make this technology scalable across a variety of imaging devices.

Title: Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions

• Authors: Angana Borah, Rada Mihalcea
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.02584
• Pdf URL: https://arxiv.org/pdf/2410.02584
• Copy Paste: [[2410.02584]] Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions(https://arxiv.org/abs/2410.02584)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) continue to evolve, they are increasingly being employed in numerous studies to simulate societies and execute diverse social tasks. However, LLMs are susceptible to societal biases due to their exposure to human-generated data. Given that LLMs are being used to gain insights into various societal aspects, it is essential to mitigate these biases. To that end, our study investigates the presence of implicit gender biases in multi-agent LLM interactions and proposes two strategies to mitigate these biases. We begin by creating a dataset of scenarios where implicit gender biases might arise, and subsequently develop a metric to assess the presence of biases. Our empirical analysis reveals that LLMs generate outputs characterized by strong implicit bias associations (>= 50\% of the time). Furthermore, these biases tend to escalate following multi-agent interactions. To mitigate them, we propose two strategies: self-reflection with in-context examples (ICE); and supervised fine-tuning. Our research demonstrates that both methods effectively mitigate implicit biases, with the ensemble of fine-tuning and self-reflection proving to be the most successful.

Title: IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengers

• Authors: Zihan Fang, Zheng Lin, Senkang Hu, Hangcheng Cao, Yiqin Deng, Xianhao Chen, Yuguang Fang
• Subjects: cs.CV, cs.AI, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.02592
• Pdf URL: https://arxiv.org/pdf/2410.02592
• Copy Paste: [[2410.02592]] IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengers(https://arxiv.org/abs/2410.02592)
• Keywords: robust, extraction
• Abstract: Recently, in-car monitoring has emerged as a promising technology for detecting early-stage abnormal status of the driver and providing timely alerts to prevent traffic accidents. Although training models with multimodal data enhances the reliability of abnormal status detection, the scarcity of labeled data and the imbalance of class distribution impede the extraction of critical abnormal state features, significantly deteriorating training performance. Furthermore, missing modalities due to environment and hardware limitations further exacerbate the challenge of abnormal status identification. More importantly, monitoring abnormal health conditions of passengers, particularly in elderly care, is of paramount importance but remains underexplored. To address these challenges, we introduce our IC3M, an efficient camera-rotation-based multimodal framework for monitoring both driver and passengers in a car. Our IC3M comprises two key modules: an adaptive threshold pseudo-labeling strategy and a missing modality reconstruction. The former customizes pseudo-labeling thresholds for different classes based on the class distribution, generating class-balanced pseudo labels to guide model training effectively, while the latter leverages crossmodality relationships learned from limited labels to accurately recover missing modalities by distribution transferring from available modalities. Extensive experimental results demonstrate that IC3M outperforms state-of-the-art benchmarks in accuracy, precision, and recall while exhibiting superior robustness under limited labeled data and severe missing modality.

Title: Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks

• Authors: Rui Hu, Yifan Zhang, Zhuoran Li, Longbo Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02596
• Pdf URL: https://arxiv.org/pdf/2410.02596
• Copy Paste: [[2410.02596]] Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks(https://arxiv.org/abs/2410.02596)
• Keywords: robust, generative
• Abstract: Generative Flow Networks (GFlowNets) are a novel class of generative models designed to sample from unnormalized distributions and have found applications in various important tasks, attracting great research interest in their training algorithms. In general, GFlowNets are trained by fitting the forward flow to the backward flow on sampled training objects. Prior work focused on the choice of training objects, parameterizations, sampling and resampling strategies, and backward policies, aiming to enhance credit assignment, exploration, or exploitation of the training process. However, the choice of regression loss, which can highly influence the exploration and exploitation behavior of the under-training policy, has been overlooked. Due to the lack of theoretical understanding for choosing an appropriate regression loss, most existing algorithms train the flow network by minimizing the squared error of the forward and backward flows in log-space, i.e., using the quadratic regression loss. In this work, we rigorously prove that distinct regression losses correspond to specific divergence measures, enabling us to design and analyze regression losses according to the desired properties of the corresponding divergence measures. Specifically, we examine two key properties: zero-forcing and zero-avoiding, where the former promotes exploitation and higher rewards, and the latter encourages exploration and enhances diversity. Based on our theoretical framework, we propose three novel regression losses, namely, Shifted-Cosh, Linex(1/2), and Linex(1). We evaluate them across three benchmarks: hyper-grid, bit-sequence generation, and molecule generation. Our proposed losses are compatible with most existing training algorithms, and significantly improve the performances of the algorithms concerning convergence speed, sample diversity, and robustness.

Title: Diffusion & Adversarial Schr\"odinger Bridges via Iterative Proportional Markovian Fitting

• Authors: Sergei Kholkin, Grigoriy Ksenofontov, David Li, Nikita Kornilov, Nikita Gushchin, Evgeny Burnaev, Alexander Korotin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02601
• Pdf URL: https://arxiv.org/pdf/2410.02601
• Copy Paste: [[2410.02601]] Diffusion & Adversarial Schr\"odinger Bridges via Iterative Proportional Markovian Fitting(https://arxiv.org/abs/2410.02601)
• Keywords: diffusion
• Abstract: The Iterative Markovian Fitting (IMF) procedure based on iterative reciprocal and Markovian projections has recently been proposed as a powerful method for solving the Schrödinger Bridge problem. However, it has been observed that for the practical implementation of this procedure, it is crucial to alternate between fitting a forward and backward time diffusion at each iteration. Such implementation is thought to be a practical heuristic, which is required to stabilize training and obtain good results in applications such as unpaired domain translation. In our work, we show that this heuristic closely connects with the pioneer approaches for the Schrödinger Bridge based on the Iterative Proportional Fitting (IPF) procedure. Namely, we find that the practical implementation of IMF is, in fact, a combination of IMF and IPF procedures, and we call this combination the Iterative Proportional Markovian Fitting (IPMF) procedure. We show both theoretically and practically that this combined IPMF procedure can converge under more general settings, thus, showing that the IPMF procedure opens a door towards developing a unified framework for solving Schrödinger Bridge problems.

Title: Agents' Room: Narrative Generation through Multi-step Collaboration

• Authors: Fantine Huot, Reinald Kim Amplayo, Jennimaria Palomaki, Alice Shoshana Jakobovits, Elizabeth Clark, Mirella Lapata
• Subjects: cs.CL, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2410.02603
• Pdf URL: https://arxiv.org/pdf/2410.02603
• Copy Paste: [[2410.02603]] Agents' Room: Narrative Generation through Multi-step Collaboration(https://arxiv.org/abs/2410.02603)
• Keywords: large language model
• Abstract: Writing compelling fiction is a multifaceted process combining elements such as crafting a plot, developing interesting characters, and using evocative language. While large language models (LLMs) show promise for story writing, they currently rely heavily on intricate prompting, which limits their use. We propose Agents' Room, a generation framework inspired by narrative theory, that decomposes narrative writing into subtasks tackled by specialized agents. To illustrate our method, we introduce Tell Me A Story, a high-quality dataset of complex writing prompts and human-written stories, and a novel evaluation framework designed specifically for assessing long narratives. We show that Agents' Room generates stories that are preferred by expert evaluators over those produced by baseline systems by leveraging collaboration and specialization to decompose the complex story writing task into tractable components. We provide extensive analysis with automated and human-based metrics of the generated output.

Title: IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?

• Authors: Akhilesh Aravapalli, Mounika Marreddy, Subba Reddy Oota, Radhika Mamidi, Manish Gupta
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02611
• Pdf URL: https://arxiv.org/pdf/2410.02611
• Copy Paste: [[2410.02611]] IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?(https://arxiv.org/abs/2410.02611)
• Keywords: robust, transformer
• Abstract: Transformer-based models have revolutionized the field of natural language processing. To understand why they perform so well and to assess their reliability, several studies have focused on questions such as: Which linguistic properties are encoded by these models, and to what extent? How robust are these models in encoding linguistic properties when faced with perturbations in the input text? However, these studies have mainly focused on BERT and the English language. In this paper, we investigate similar questions regarding encoding capability and robustness for 8 linguistic properties across 13 different perturbations in 6 Indic languages, using 9 multilingual Transformer models (7 universal and 2 Indic-specific). To conduct this study, we introduce a novel multilingual benchmark dataset, IndicSentEval, containing approximately $\sim$47K sentences. Surprisingly, our probing analysis of surface, syntactic, and semantic properties reveals that while almost all multilingual models demonstrate consistent encoding performance for English, they show mixed results for Indic languages. As expected, Indic-specific multilingual models capture linguistic properties in Indic languages better than universal models. Intriguingly, universal models broadly exhibit better robustness compared to Indic-specific models, particularly under perturbations such as dropping both nouns and verbs, dropping only verbs, or keeping only nouns. Overall, this study provides valuable insights into probing and perturbation-specific strengths and weaknesses of popular multilingual Transformer-based models for different Indic languages. We make our code and dataset publicly available [this https URL}].

Title: NL-Eye: Abductive NLI for Images

• Authors: Mor Ventura, Michael Toker, Nitay Calderon, Zorik Gekhman, Yonatan Bitton, Roi Reichart
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02613
• Pdf URL: https://arxiv.org/pdf/2410.02613
• Copy Paste: [[2410.02613]] NL-Eye: Abductive NLI for Images(https://arxiv.org/abs/2410.02613)
• Keywords: robust
• Abstract: Will a Visual Language Model (VLM)-based bot warn us about slipping if it detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet their ability to infer outcomes and causes remains underexplored. To address this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual abductive reasoning skills. NL-Eye adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. The data curation process involved two steps - writing textual descriptions and generating images using text-to-image models, both requiring substantial human involvement to ensure high-quality and challenging scenes. Our experiments show that VLMs struggle significantly on NL-Eye, often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality. This demonstrates a deficiency in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a crucial step toward developing VLMs capable of robust multimodal reasoning for real-world applications, including accident-prevention bots and generated video verification.

Title: LoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model

• Authors: Duy M. H. Nguyen, Nghiem T. Diep, Trung Q. Nguyen, Hoang-Bao Le, Tai Nguyen, Tien Nguyen, TrungTin Nguyen, Nhat Ho, Pengtao Xie, Roger Wattenhofer, James Zhou, Daniel Sonntag, Mathias Niepert
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02615
• Pdf URL: https://arxiv.org/pdf/2410.02615
• Copy Paste: [[2410.02615]] LoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model(https://arxiv.org/abs/2410.02615)
• Keywords: large language model
• Abstract: State-of-the-art medical multi-modal large language models (med-MLLM), like LLaVA-Med or BioMedGPT, leverage instruction-following data in pre-training. However, those models primarily focus on scaling the model size and data volume to boost performance while mainly relying on the autoregressive learning objectives. Surprisingly, we reveal that such learning schemes might result in a weak alignment between vision and language modalities, making these models highly reliant on extensive pre-training datasets - a significant challenge in medical domains due to the expensive and time-consuming nature of curating high-quality instruction-following instances. We address this with LoGra-Med, a new multi-graph alignment algorithm that enforces triplet correlations across image modalities, conversation-based descriptions, and extended captions. This helps the model capture contextual meaning, handle linguistic variability, and build cross-modal associations between visuals and text. To scale our approach, we designed an efficient end-to-end learning scheme using black-box gradient estimation, enabling faster LLaMa 7B training. Our results show LoGra-Med matches LLAVA-Med performance on 600K image-text pairs for Medical VQA and significantly outperforms it when trained on 10% of the data. For example, on VQA-RAD, we exceed LLAVA-Med by 20.13% and nearly match the 100% pre-training score (72.52% vs. 72.64%). We also surpass SOTA methods like BiomedGPT on visual chatbots and RadFM on zero-shot image classification with VQA, highlighting the effectiveness of multi-graph alignment.

Title: Diss-l-ECT: Dissecting Graph Data with local Euler Characteristic Transforms

• Authors: Julius von Rohrscheidt, Bastian Rieck
• Subjects: cs.LG, math.AT
• Abstract URL: https://arxiv.org/abs/2410.02622
• Pdf URL: https://arxiv.org/pdf/2410.02622
• Copy Paste: [[2410.02622]] Diss-l-ECT: Dissecting Graph Data with local Euler Characteristic Transforms(https://arxiv.org/abs/2410.02622)
• Keywords: interpretability
• Abstract: The Euler Characteristic Transform (ECT) is an efficiently-computable geometrical-topological invariant that characterizes the global shape of data. In this paper, we introduce the Local Euler Characteristic Transform ($\ell$-ECT), a novel extension of the ECT particularly designed to enhance expressivity and interpretability in graph representation learning. Unlike traditional Graph Neural Networks (GNNs), which may lose critical local details through aggregation, the $\ell$-ECT provides a lossless representation of local neighborhoods. This approach addresses key limitations in GNNs by preserving nuanced local structures while maintaining global interpretability. Moreover, we construct a rotation-invariant metric based on $\ell$-ECTs for spatial alignment of data spaces. Our method exhibits superior performance than standard GNNs on a variety of node classification tasks, particularly in graphs with high heterophily.

Title: Metrics Revolutions: Groundbreaking Insights into the Implementation of Metrics for Biomedical Image Segmentation

• Authors: Gašper Podobnik, Tomaž Vrtovec
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02630
• Pdf URL: https://arxiv.org/pdf/2410.02630
• Copy Paste: [[2410.02630]] Metrics Revolutions: Groundbreaking Insights into the Implementation of Metrics for Biomedical Image Segmentation(https://arxiv.org/abs/2410.02630)
• Keywords: segmentation
• Abstract: The evaluation of segmentation performance is a common task in biomedical image analysis, with its importance emphasized in the recently released metrics selection guidelines and computing frameworks. To quantitatively evaluate the alignment of two segmentations, researchers commonly resort to counting metrics, such as the Dice similarity coefficient, or distance-based metrics, such as the Hausdorff distance, which are usually computed by publicly available open-source tools with an inherent assumption that these tools provide consistent results. In this study we questioned this assumption, and performed a systematic implementation analysis along with quantitative experiments on real-world clinical data to compare 11 open-source tools for distance-based metrics computation against our highly accurate mesh-based reference implementation. The results revealed that statistically significant differences among all open-source tools are both surprising and concerning, since they question the validity of existing studies. Besides identifying the main sources of variation, we also provide recommendations for distance-based metrics computation.

Title: Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning

• Authors: Tianxiang Hu, Pei Zhang, Baosong Yang, Jun Xie, Derek F. Wong, Rui Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02631
• Pdf URL: https://arxiv.org/pdf/2410.02631
• Copy Paste: [[2410.02631]] Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning(https://arxiv.org/abs/2410.02631)
• Keywords: robust, large language model
• Abstract: Achieving consistent high-quality machine translation (MT) across diverse domains remains a significant challenge, primarily due to the limited and imbalanced parallel training data available in various domains. While large language models (LLMs) have demonstrated impressive general understanding and generation abilities, their potential in multi-domain MT is under-explored. We establish a comprehensive benchmark for multi-domain translation, featuring 25 German$\Leftrightarrow$English and 22 Chinese$\Leftrightarrow$English test sets respectively covering 15 domains. Our evaluation of prominent LLMs reveals a discernible performance gap against traditional MT systems, highlighting domain overfitting and catastrophic forgetting issues after fine-tuning on domain-limited corpora. To mitigate this, we propose a domain Chain of Thought (CoT) fine-tuning technique that utilizes the intrinsic multi-domain intelligence of LLMs to improve translation performance. This method inspires the LLM to perceive domain information from the source text, which then serves as a helpful hint to guide the translation process. Despite being trained on a small dataset of four domains, our CoT fine-tune approach achieves notable enhancements in translation accuracy and domain robustness than traditional fine-tuning, as evidenced by an average 1.53 BLEU score increase in over 20 German$\rightarrow$English distinct out-of-domain tests.

Title: Spatial-Temporal Multi-Cuts for Online Multiple-Camera Vehicle Tracking

• Authors: Fabian Herzog, Johannes Gilg, Philipp Wolters, Torben Teepe, Gerhard Rigoll
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02638
• Pdf URL: https://arxiv.org/pdf/2410.02638
• Copy Paste: [[2410.02638]] Spatial-Temporal Multi-Cuts for Online Multiple-Camera Vehicle Tracking(https://arxiv.org/abs/2410.02638)
• Keywords: extraction
• Abstract: Accurate online multiple-camera vehicle tracking is essential for intelligent transportation systems, autonomous driving, and smart city applications. Like single-camera multiple-object tracking, it is commonly formulated as a graph problem of tracking-by-detection. Within this framework, existing online methods usually consist of two-stage procedures that cluster temporally first, then spatially, or vice versa. This is computationally expensive and prone to error accumulation. We introduce a graph representation that allows spatial-temporal clustering in a single, combined step: New detections are spatially and temporally connected with existing clusters. By keeping sparse appearance and positional cues of all detections in a cluster, our method can compare clusters based on the strongest available evidence. The final tracks are obtained online using a simple multicut assignment procedure. Our method does not require any training on the target scene, pre-extraction of single-camera tracks, or additional annotations. Notably, we outperform the online state-of-the-art on the CityFlow dataset in terms of IDF1 by more than 14%, and on the Synthehicle dataset by more than 25%, respectively. The code is publicly available.

Title: Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers

• Authors: Shijie Chen, Bernal Jiménez Gutiérrez, Yu Su
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.02642
• Pdf URL: https://arxiv.org/pdf/2410.02642
• Copy Paste: [[2410.02642]] Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers(https://arxiv.org/abs/2410.02642)
• Keywords: generative, large language model
• Abstract: Information retrieval (IR) systems have played a vital role in modern digital life and have cemented their continued usefulness in this new era of generative AI via retrieval-augmented generation. With strong language processing capabilities and remarkable versatility, large language models (LLMs) have become popular choices for zero-shot re-ranking in IR systems. So far, LLM-based re-ranking methods rely on strong generative capabilities, which restricts their use to either specialized or powerful proprietary models. Given these restrictions, we ask: is autoregressive generation necessary and optimal for LLMs to perform re-ranking? We hypothesize that there are abundant signals relevant to re-ranking within LLMs that might not be used to their full potential via generation. To more directly leverage such signals, we propose in-context re-ranking (ICR), a novel method that leverages the change in attention pattern caused by the search query for accurate and efficient re-ranking. To mitigate the intrinsic biases in LLMs, we propose a calibration method using a content-free query. Due to the absence of generation, ICR only requires two ($O(1)$) forward passes to re-rank $N$ documents, making it substantially more efficient than generative re-ranking methods that require at least $O(N)$ forward passes. Our novel design also enables ICR to be applied to any LLM without specialized training while guaranteeing a well-formed ranking. Extensive experiments with two popular open-weight LLMs on standard single-hop and multi-hop information retrieval benchmarks show that ICR outperforms RankGPT while cutting the latency by more than 60% in practice. Through detailed analyses, we show that ICR's performance is specially strong on tasks that require more complex re-ranking signals. Our findings call for further exploration on novel ways of utilizing open-weight LLMs beyond text generation.

Title: Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

• Authors: Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, Yongfeng Zhang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02644
• Pdf URL: https://arxiv.org/pdf/2410.02644
• Copy Paste: [[2410.02644]] Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents(https://arxiv.org/abs/2410.02644)
• Keywords: security, defense, attack, large language model
• Abstract: Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 23 different types of attack/defense methods, and 8 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, a mixed attack, and 10 corresponding defenses across 13 LLM backbones with nearly 90,000 testing cases in total. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30\%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. Our code can be found at this https URL.

Title: Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection

• Authors: Song Li, Yang Tan, Song Ke, Liang Hong, Bingxin Zhou
• Subjects: cs.LG, cs.CL, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.02647
• Pdf URL: https://arxiv.org/pdf/2410.02647
• Copy Paste: [[2410.02647]] Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection(https://arxiv.org/abs/2410.02647)
• Keywords: protect
• Abstract: Immunogenicity prediction is a central topic in reverse vaccinology for finding candidate vaccines that can trigger protective immune responses. Existing approaches typically rely on highly compressed features and simple model architectures, leading to limited prediction accuracy and poor generalizability. To address these challenges, we introduce ProVaccine, a novel deep learning solution with a dual attention mechanism that integrates pre-trained latent vector representations of protein sequences and structures. We also compile the most comprehensive immunogenicity dataset to date, encompassing over 9,500 antigen sequences, structures, and immunogenicity labels from bacteria, viruses, and tumors. Extensive experiments demonstrate that ProVaccine outperforms existing methods across a wide range of evaluation metrics. Furthermore, we establish a post-hoc validation protocol to assess the practical significance of deep learning models in tackling vaccine design challenges. Our work provides an effective tool for vaccine design and sets valuable benchmarks for future research.

Title: Undesirable Memorization in Large Language Models: A Survey

• Authors: Ali Satvaty, Suzan Verberne, Fatih Turkmen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02650
• Pdf URL: https://arxiv.org/pdf/2410.02650
• Copy Paste: [[2410.02650]] Undesirable Memorization in Large Language Models: A Survey(https://arxiv.org/abs/2410.02650)
• Keywords: security, privacy, attack, diffusion, large language model
• Abstract: While recent research increasingly showcases the remarkable capabilities of Large Language Models (LLMs), it's vital to confront their hidden pitfalls. Among these challenges, the issue of memorization stands out, posing significant ethical and legal risks. In this paper, we presents a Systematization of Knowledge (SoK) on the topic of memorization in LLMs. Memorization is the effect that a model tends to store and reproduce phrases or passages from the training data and has been shown to be the fundamental issue to various privacy and security attacks against LLMs. We begin by providing an overview of the literature on the memorization, exploring it across five key dimensions: intentionality, degree, retrievability, abstraction, and transparency. Next, we discuss the metrics and methods used to measure memorization, followed by an analysis of the factors that contribute to memorization phenomenon. We then examine how memorization manifests itself in specific model architectures and explore strategies for mitigating these effects. We conclude our overview by identifying potential research topics for the near future: to develop methods for balancing performance and privacy in LLMs, and the analysis of memorization in specific contexts, including conversational agents, retrieval-augmented generation, multilingual language models, and diffusion language models.

Title: Measuring and Improving Persuasiveness of Generative Models

• Authors: Somesh Singh, Yaman K Singla, Harini SI, Balaji Krishnamurthy
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02653
• Pdf URL: https://arxiv.org/pdf/2410.02653
• Copy Paste: [[2410.02653]] Measuring and Improving Persuasiveness of Generative Models(https://arxiv.org/abs/2410.02653)
• Keywords: generative
• Abstract: LLMs are increasingly being used in workflows involving generating content to be consumed by humans (e.g., marketing) and also in directly interacting with humans (e.g., through chatbots). The development of such systems that are capable of generating verifiably persuasive messages presents both opportunities and challenges for society. On the one hand, such systems could positively impact domains like advertising and social good, such as addressing drug addiction, and on the other, they could be misused for spreading misinformation and shaping political opinions. To channel LLMs' impact on society, we need to develop systems to measure and benchmark their persuasiveness. With this motivation, we introduce PersuasionBench and PersuasionArena, the first large-scale benchmark and arena containing a battery of tasks to measure the persuasion ability of generative models automatically. We investigate to what extent LLMs know and leverage linguistic patterns that can help them generate more persuasive language. Our findings indicate that the persuasiveness of LLMs correlates positively with model size, but smaller models can also be made to have a higher persuasiveness than much larger models. Notably, targeted training using synthetic and natural datasets significantly enhances smaller models' persuasive capabilities, challenging scale-dependent assumptions. Our findings carry key implications for both model developers and policymakers. For instance, while the EU AI Act and California's SB-1047 aim to regulate AI models based on the number of floating point operations, we demonstrate that simple metrics like this alone fail to capture the full scope of AI's societal impact. We invite the community to explore and contribute to PersuasionArena and PersuasionBench, available at this https URL, to advance our understanding of AI-driven persuasion and its societal implications.

Title: Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems

• Authors: Hunter Heidenreich, Pantelis R. Vlachas, etros Koumoutsakos
• Subjects: cs.LG, nlin.CD, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2410.02654
• Pdf URL: https://arxiv.org/pdf/2410.02654
• Copy Paste: [[2410.02654]] Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems(https://arxiv.org/abs/2410.02654)
• Keywords: transformer
• Abstract: Machine learning architectures, including transformers and recurrent neural networks (RNNs) have revolutionized forecasting in applications ranging from text processing to extreme weather. Notably, advanced network architectures, tuned for applications such as natural language processing, are transferable to other tasks such as spatiotemporal forecasting tasks. However, there is a scarcity of ablation studies to illustrate the key components that enable this forecasting accuracy. The absence of such studies, although explainable due to the associated computational cost, intensifies the belief that these models ought to be considered as black boxes. In this work, we decompose the key architectural components of the most powerful neural architectures, namely gating and recurrence in RNNs, and attention mechanisms in transformers. Then, we synthesize and build novel hybrid architectures from the standard blocks, performing ablation studies to identify which mechanisms are effective for each task. The importance of considering these components as hyper-parameters that can augment the standard architectures is exhibited on various forecasting datasets, from the spatiotemporal chaotic dynamics of the multiscale Lorenz 96 system, the Kuramoto-Sivashinsky equation, as well as standard real world time-series benchmarks. A key finding is that neural gating and attention improves the performance of all standard RNNs in most tasks, while the addition of a notion of recurrence in transformers is detrimental. Furthermore, our study reveals that a novel, sparsely used, architecture which integrates Recurrent Highway Networks with neural gating and attention mechanisms, emerges as the best performing architecture in high-dimensional spatiotemporal forecasting of dynamical systems.

Title: Scalable Simulation-free Entropic Unbalanced Optimal Transport

• Authors: Jaemoo Choi, Jaewoong Choi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02656
• Pdf URL: https://arxiv.org/pdf/2410.02656
• Copy Paste: [[2410.02656]] Scalable Simulation-free Entropic Unbalanced Optimal Transport(https://arxiv.org/abs/2410.02656)
• Keywords: generative
• Abstract: The Optimal Transport (OT) problem investigates a transport map that connects two distributions while minimizing a given cost function. Finding such a transport map has diverse applications in machine learning, such as generative modeling and image-to-image translation. In this paper, we introduce a scalable and simulation-free approach for solving the Entropic Unbalanced Optimal Transport (EUOT) problem. We derive the dynamical form of this EUOT problem, which is a generalization of the Schrödinger bridges (SB) problem. Based on this, we derive dual formulation and optimality conditions of the EUOT problem from the stochastic optimal control interpretation. By leveraging these properties, we propose a simulation-free algorithm to solve EUOT, called Simulation-free EUOT (SF-EUOT). While existing SB models require expensive simulation costs during training and evaluation, our model achieves simulation-free training and one-step generation by utilizing the reciprocal property. Our model demonstrates significantly improved scalability in generative modeling and image-to-image translation tasks compared to previous SB methods.

Title: Hate Personified: Investigating the role of LLMs in content moderation

• Authors: Sarah Masud, Sahajpreet Singh, Viktor Hangya, Alexander Fraser, Tanmoy Chakraborty
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.02657
• Pdf URL: https://arxiv.org/pdf/2410.02657
• Copy Paste: [[2410.02657]] Hate Personified: Investigating the role of LLMs in content moderation(https://arxiv.org/abs/2410.02657)
• Keywords: large language model
• Abstract: For subjective tasks such as hate detection, where people perceive hate differently, the Large Language Model's (LLM) ability to represent diverse groups is unclear. By including additional context in prompts, we comprehensively analyze LLM's sensitivity to geographical priming, persona attributes, and numerical information to assess how well the needs of various groups are reflected. Our findings on two LLMs, five languages, and six datasets reveal that mimicking persona-based attributes leads to annotation variability. Meanwhile, incorporating geographical signals leads to better regional alignment. We also find that the LLMs are sensitive to numerical anchors, indicating the ability to leverage community-based flagging efforts and exposure to adversaries. Our work provides preliminary guidelines and highlights the nuances of applying LLMs in culturally sensitive cases.

Title: How to Train Long-Context Language Models (Effectively)

• Authors: Tianyu Gao, Alexander Wettig, Howard Yen, Danqi Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02660
• Pdf URL: https://arxiv.org/pdf/2410.02660
• Copy Paste: [[2410.02660]] How to Train Long-Context Language Models (Effectively)(https://arxiv.org/abs/2410.02660)
• Keywords: robust
• Abstract: We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. We first establish a reliable evaluation protocol to guide model development -- Instead of perplexity or simple needle-in-a-haystack (NIAH) tests, we use a broad set of long-context tasks, and we evaluate models after SFT with instruction data as this better reveals long-context abilities. Supported by our robust evaluations, we run thorough experiments to decide the data mix for continued pre-training, the instruction tuning dataset, and many other design choices. We find that (1) code repositories and books are excellent sources of long data, but it is crucial to combine them with high-quality short data; (2) training with a sequence length beyond the evaluation length boosts long-context performance; (3) for SFT, using only short instruction datasets yields strong performance on long-context tasks. Our final model, ProLong-8B, which is initialized from Llama-3 and trained on 40B tokens, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K. ProLong outperforms Llama-3.18B-Instruct on the majority of long-context tasks despite having seen only 5% as many tokens during long-context training. Additionally, ProLong can effectively process up to 512K tokens, one of the longest context windows of publicly available LMs.

Title: AlphaIntegrator: Transformer Action Search for Symbolic Integration Proofs

• Authors: Mert Ünsal, Timon Gehr, Martin Vechev
• Subjects: cs.LG, cs.AI, cs.MS, cs.SC
• Abstract URL: https://arxiv.org/abs/2410.02666
• Pdf URL: https://arxiv.org/pdf/2410.02666
• Copy Paste: [[2410.02666]] AlphaIntegrator: Transformer Action Search for Symbolic Integration Proofs(https://arxiv.org/abs/2410.02666)
• Keywords: transformer
• Abstract: We present the first correct-by-construction learning-based system for step-by-step mathematical integration. The key idea is to learn a policy, represented by a GPT transformer model, which guides the search for the right mathematical integration rule, to be carried out by a symbolic solver. Concretely, we introduce a symbolic engine with axiomatically correct actions on mathematical expressions, as well as the first dataset for step-by-step integration. Our GPT-style transformer model, trained on this synthetic data, demonstrates strong generalization by surpassing its own data generator in accuracy and efficiency, using 50% fewer search steps. Our experimental results with SoTA LLMs also demonstrate that the standard approach of fine-tuning LLMs on a set of question-answer pairs is insufficient for solving this mathematical task. This motivates the importance of discovering creative methods for combining LLMs with symbolic reasoning engines, of which our work is an instance.

Title: GUD: Generation with Unified Diffusion

• Authors: Mathis Gerdes, Max Welling, Miranda C. N. Cheng
• Subjects: cs.LG, hep-th, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02667
• Pdf URL: https://arxiv.org/pdf/2410.02667
• Copy Paste: [[2410.02667]] GUD: Generation with Unified Diffusion(https://arxiv.org/abs/2410.02667)
• Keywords: diffusion, generative
• Abstract: Diffusion generative models transform noise into data by inverting a process that progressively adds noise to data samples. Inspired by concepts from the renormalization group in physics, which analyzes systems across different scales, we revisit diffusion models by exploring three key design aspects: 1) the choice of representation in which the diffusion process operates (e.g. pixel-, PCA-, Fourier-, or wavelet-basis), 2) the prior distribution that data is transformed into during diffusion (e.g. Gaussian with covariance $\Sigma$), and 3) the scheduling of noise levels applied separately to different parts of the data, captured by a component-wise noise schedule. Incorporating the flexibility in these choices, we develop a unified framework for diffusion generative models with greatly enhanced design freedom. In particular, we introduce soft-conditioning models that smoothly interpolate between standard diffusion models and autoregressive models (in any basis), conceptually bridging these two approaches. Our framework opens up a wide design space which may lead to more efficient training and data generation, and paves the way to novel architectures integrating different generative approaches and generation tasks.

Title: FAN: Fourier Analysis Networks

• Authors: Yihong Dong, Ge Li, Yongding Tao, Xue Jiang, Kechi Zhang, Jia Li, Jing Su, Jun Zhang, Jingjing Xu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02675
• Pdf URL: https://arxiv.org/pdf/2410.02675
• Copy Paste: [[2410.02675]] FAN: Fourier Analysis Networks(https://arxiv.org/abs/2410.02675)
• Keywords: transformer
• Abstract: Despite the remarkable success achieved by neural networks, particularly those represented by MLP and Transformer, we reveal that they exhibit potential flaws in the modeling and reasoning of periodicity, i.e., they tend to memorize the periodic data rather than genuinely understanding the underlying principles of periodicity. However, periodicity is a crucial trait in various forms of reasoning and generalization, underpinning predictability across natural and engineered systems through recurring patterns in observations. In this paper, we propose FAN, a novel network architecture based on Fourier Analysis, which empowers the ability to efficiently model and reason about periodic phenomena. By introducing Fourier Series, the periodicity is naturally integrated into the structure and computational processes of the neural network, thus achieving a more accurate expression and prediction of periodic patterns. As a promising substitute to multi-layer perceptron (MLP), FAN can seamlessly replace MLP in various models with fewer parameters and FLOPs. Through extensive experiments, we demonstrate the effectiveness of FAN in modeling and reasoning about periodic functions, and the superiority and generalizability of FAN across a range of real-world tasks, including symbolic formula representation, time series forecasting, and language modeling.

Title: CulturalBench: a Robust, Diverse and Challenging Benchmark on Measuring the (Lack of) Cultural Knowledge of LLMs

• Authors: Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02677
• Pdf URL: https://arxiv.org/pdf/2410.02677
• Copy Paste: [[2410.02677]] CulturalBench: a Robust, Diverse and Challenging Benchmark on Measuring the (Lack of) Cultural Knowledge of LLMs(https://arxiv.org/abs/2410.02677)
• Keywords: robust, large language model
• Abstract: To make large language models (LLMs) more helpful across diverse cultures, it is essential to have effective cultural knowledge benchmarks to measure and track our progress. Effective benchmarks need to be robust, diverse, and challenging. We introduce CulturalBench: a set of 1,227 human-written and human-verified questions for effectively assessing LLMs' cultural knowledge, covering 45 global regions including the underrepresented ones like Bangladesh, Zimbabwe, and Peru. Questions - each verified by five independent annotators - span 17 diverse topics ranging from food preferences to greeting etiquettes. We evaluate models on two setups: CulturalBench-Easy and CulturalBench-Hard which share the same questions but asked differently. We find that LLMs are sensitive to such difference in setups (e.g., GPT-4o with 27.3% difference). Compared to human performance (92.6% accuracy), CulturalBench-Hard is more challenging for frontier LLMs with the best performing model (GPT-4o) at only 61.5% and the worst (Llama3-8b) at 21.4%. Moreover, we find that LLMs often struggle with tricky questions that have multiple correct answers (e.g., What utensils do the Chinese usually use?), revealing a tendency to converge to a single answer. Our results also indicate that OpenAI GPT-4o substantially outperform other proprietary and open source models in questions related to all but one region (Oceania). Nonetheless, all models consistently underperform on questions related to South America and the Middle East.

Title: Distilling an End-to-End Voice Assistant Without Instruction Training Data

• Authors: William Held, Ella Li, Michael Ryan, Weiyan Shi, Yanzhe Zhang, Diyi Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02678
• Pdf URL: https://arxiv.org/pdf/2410.02678
• Copy Paste: [[2410.02678]] Distilling an End-to-End Voice Assistant Without Instruction Training Data(https://arxiv.org/abs/2410.02678)
• Keywords: large language model
• Abstract: Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT) have led to models ``forgetting" capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72\% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using $>$100x less training compute.

Title: HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router

• Authors: Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Ruibin Yuan, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02684
• Pdf URL: https://arxiv.org/pdf/2410.02684
• Copy Paste: [[2410.02684]] HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router(https://arxiv.org/abs/2410.02684)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) grow increasingly powerful, ensuring their safety and alignment with human values remains a critical challenge. Ideally, LLMs should provide informative responses while avoiding the disclosure of harmful or sensitive information. However, current alignment approaches, which rely heavily on refusal strategies, such as training models to completely reject harmful prompts or applying coarse filters are limited by their binary nature. These methods either fully deny access to information or grant it without sufficient nuance, leading to overly cautious responses or failures to detect subtle harmful content. For example, LLMs may refuse to provide basic, public information about medication due to misuse concerns. Moreover, these refusal-based methods struggle to handle mixed-content scenarios and lack the ability to adapt to context-dependent sensitivities, which can result in over-censorship of benign content. To overcome these challenges, we introduce HiddenGuard, a novel framework for fine-grained, safe generation in LLMs. HiddenGuard incorporates Prism (rePresentation Router for In-Stream Moderation), which operates alongside the LLM to enable real-time, token-level detection and redaction of harmful content by leveraging intermediate hidden states. This fine-grained approach allows for more nuanced, context-aware moderation, enabling the model to generate informative responses while selectively redacting or replacing sensitive information, rather than outright refusal. We also contribute a comprehensive dataset with token-level fine-grained annotations of potentially harmful information across diverse contexts. Our experiments demonstrate that HiddenGuard achieves over 90% in F1 score for detecting and redacting harmful content while preserving the overall utility and informativeness of the model's responses.

Title: Discovering Clues of Spoofed LM Watermarks

• Authors: Thibaud Gloaguen, Nikola Jovanović, Robin Staab, Martin Vechev
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02693
• Pdf URL: https://arxiv.org/pdf/2410.02693
• Copy Paste: [[2410.02693]] Discovering Clues of Spoofed LM Watermarks(https://arxiv.org/abs/2410.02693)
• Keywords: attack, watermark
• Abstract: LLM watermarks stand out as a promising way to attribute ownership of LLM-generated text. One threat to watermark credibility comes from spoofing attacks, where an unauthorized third party forges the watermark, enabling it to falsely attribute arbitrary texts to a particular LLM. While recent works have demonstrated that state-of-the-art schemes are in fact vulnerable to spoofing, they lack deeper qualitative analysis of the texts produced by spoofing methods. In this work, we for the first time reveal that there are observable differences between genuine and spoofed watermark texts. Namely, we show that regardless of their underlying approach, all current spoofing methods consistently leave observable artifacts in spoofed texts, indicative of watermark forgery. We build upon these findings to propose rigorous statistical tests that reliably reveal the presence of such artifacts, effectively discovering that a watermark was spoofed. Our experimental evaluation shows high test power across all current spoofing methods, providing insights into their fundamental limitations, and suggesting a way to mitigate this threat.

Title: HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly

• Authors: Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izasak, Moshe Wasserblat, Danqi Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02694
• Pdf URL: https://arxiv.org/pdf/2410.02694
• Copy Paste: [[2410.02694]] HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly(https://arxiv.org/abs/2410.02694)
• Keywords: robust
• Abstract: There have been many benchmarks for evaluating long-context language models (LCLMs), but developers often rely on synthetic tasks like needle-in-a-haystack (NIAH) or arbitrary subsets of tasks. It remains unclear whether they translate to the diverse downstream applications of LCLMs, and the inconsistency further complicates model comparison. We investigate the underlying reasons behind current practices and find that existing benchmarks often provide noisy signals due to low coverage of applications, insufficient lengths, unreliable metrics, and incompatibility with base models. In this work, we present HELMET (How to Evaluate Long-context Models Effectively and Thoroughly), a comprehensive benchmark encompassing seven diverse, application-centric categories. We also address many issues in previous benchmarks by adding controllable lengths up to 128k tokens, model-based evaluation for reliable metrics, and few-shot prompting for robustly evaluating base models. Consequently, we demonstrate that HELMET offers more reliable and consistent rankings of frontier LCLMs. Through a comprehensive study of 51 LCLMs, we find that (1) synthetic tasks like NIAH are not good predictors of downstream performance; (2) the diverse categories in HELMET exhibit distinct trends and low correlation with each other; and (3) while most LCLMs achieve perfect NIAH scores, open-source models significantly lag behind closed ones when the task requires full-context reasoning or following complex instructions -- the gap widens with increased lengths. Finally, we recommend using our RAG tasks for fast model development, as they are easy to run and more predictive of other downstream performance; ultimately, we advocate for a holistic evaluation across diverse tasks.

Title: Lie Algebra Canonicalization: Equivariant Neural Operators under arbitrary Lie Groups

• Authors: Zakhar Shumaylov, Peter Zaika, James Rowbottom, Ferdia Sherry, Melanie Weber, Carola-Bibiane Schönlieb
• Subjects: cs.LG, cs.CV, math.NA
• Abstract URL: https://arxiv.org/abs/2410.02698
• Pdf URL: https://arxiv.org/pdf/2410.02698
• Copy Paste: [[2410.02698]] Lie Algebra Canonicalization: Equivariant Neural Operators under arbitrary Lie Groups(https://arxiv.org/abs/2410.02698)
• Keywords: robust
• Abstract: The quest for robust and generalizable machine learning models has driven recent interest in exploiting symmetries through equivariant neural networks. In the context of PDE solvers, recent works have shown that Lie point symmetries can be a useful inductive bias for Physics-Informed Neural Networks (PINNs) through data and loss augmentation. Despite this, directly enforcing equivariance within the model architecture for these problems remains elusive. This is because many PDEs admit non-compact symmetry groups, oftentimes not studied beyond their infinitesimal generators, making them incompatible with most existing equivariant architectures. In this work, we propose Lie aLgebrA Canonicalization (LieLAC), a novel approach that exploits only the action of infinitesimal generators of the symmetry group, circumventing the need for knowledge of the full group structure. To achieve this, we address existing theoretical issues in the canonicalization literature, establishing connections with frame averaging in the case of continuous non-compact groups. Operating within the framework of canonicalization, LieLAC can easily be integrated with unconstrained pre-trained models, transforming inputs to a canonical form before feeding them into the existing model, effectively aligning the input for model inference according to allowed symmetries. LieLAC utilizes standard Lie group descent schemes, achieving equivariance in pre-trained models. Finally, we showcase LieLAC's efficacy on tasks of invariant image classification and Lie point symmetry equivariant neural PDE solvers using pre-trained models.

Title: Selective Attention Improves Transformer

• Authors: Yaniv Leviathan, Matan Kalman, Yossi Matias
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02703
• Pdf URL: https://arxiv.org/pdf/2410.02703
• Copy Paste: [[2410.02703]] Selective Attention Improves Transformer(https://arxiv.org/abs/2410.02703)
• Keywords: transformer
• Abstract: Unneeded elements in the attention's context degrade performance. We introduce Selective Attention, a simple parameter-free change to the standard attention mechanism which reduces attention to unneeded elements. Selective attention improves language modeling performance in a variety of model sizes and context lengths. For example, a range of transformers trained with the language modeling objective on C4 with selective attention perform equivalently to standard transformers with ~2X more heads and parameters in their attention modules. Selective attention also allows decreasing the size of the attention's context buffer, leading to meaningful reductions in the memory and compute requirements during inference. For example, transformers with 100M parameters trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and 47X less memory for their attention module, respectively, when equipped with selective attention, as those without selective attention, with the same validation perplexity.

Title: ControlAR: Controllable Image Generation with Autoregressive Models

• Authors: Zongming Li, Tianheng Cheng, Shoufa Chen, Peize Sun, Haocheng Shen, Longjin Ran, Xiaoxin Chen, Wenyu Liu, Xinggang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02705
• Pdf URL: https://arxiv.org/pdf/2410.02705
• Copy Paste: [[2410.02705]] ControlAR: Controllable Image Generation with Autoregressive Models(https://arxiv.org/abs/2410.02705)
• Keywords: diffusion, large language model, segmentation
• Abstract: Autoregressive (AR) models have reformulated image generation as next-token prediction, demonstrating remarkable potential and emerging as strong competitors to diffusion models. However, control-to-image generation, akin to ControlNet, remains largely unexplored within AR models. Although a natural approach, inspired by advancements in Large Language Models, is to tokenize control images into tokens and prefill them into the autoregressive model before decoding image tokens, it still falls short in generation quality compared to ControlNet and suffers from inefficiency. To this end, we introduce ControlAR, an efficient and effective framework for integrating spatial controls into autoregressive image generation models. Firstly, we explore control encoding for AR models and propose a lightweight control encoder to transform spatial inputs (e.g., canny edges or depth maps) into control tokens. Then ControlAR exploits the conditional decoding method to generate the next image token conditioned on the per-token fusion between control and image tokens, similar to positional encodings. Compared to prefilling tokens, using conditional decoding significantly strengthens the control capability of AR models but also maintains the model's efficiency. Furthermore, the proposed ControlAR surprisingly empowers AR models with arbitrary-resolution image generation via conditional decoding and specific controls. Extensive experiments can demonstrate the controllability of the proposed ControlAR for the autoregressive control-to-image generation across diverse inputs, including edges, depths, and segmentation masks. Furthermore, both quantitative and qualitative results indicate that ControlAR surpasses previous state-of-the-art controllable diffusion models, e.g., ControlNet++. Code, models, and demo will soon be available at this https URL.

Title: LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

• Authors: Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, Yonatan Belinkov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02707
• Pdf URL: https://arxiv.org/pdf/2410.02707
• Copy Paste: [[2410.02707]] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations(https://arxiv.org/abs/2410.02707)
• Keywords: large language model
• Abstract: Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that -- contrary to prior claims -- truthfulness encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between LLMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation.

Title: SteerDiff: Steering towards Safe Text-to-Image Diffusion Models

• Authors: Hongxiang Zhang, Yifeng He, Hao Chen
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02710
• Pdf URL: https://arxiv.org/pdf/2410.02710
• Copy Paste: [[2410.02710]] SteerDiff: Steering towards Safe Text-to-Image Diffusion Models(https://arxiv.org/abs/2410.02710)
• Keywords: attack, robust, diffusion
• Abstract: Text-to-image (T2I) diffusion models have drawn attention for their ability to generate high-quality images with precise text alignment. However, these models can also be misused to produce inappropriate content. Existing safety measures, which typically rely on text classifiers or ControlNet-like approaches, are often insufficient. Traditional text classifiers rely on large-scale labeled datasets and can be easily bypassed by rephrasing. As diffusion models continue to scale, fine-tuning these safeguards becomes increasingly challenging and lacks flexibility. Recent red-teaming attack researches further underscore the need for a new paradigm to prevent the generation of inappropriate content. In this paper, we introduce SteerDiff, a lightweight adaptor module designed to act as an intermediary between user input and the diffusion model, ensuring that generated images adhere to ethical and safety standards with little to no impact on usability. SteerDiff identifies and manipulates inappropriate concepts within the text embedding space to guide the model away from harmful outputs. We conduct extensive experiments across various concept unlearning tasks to evaluate the effectiveness of our approach. Furthermore, we benchmark SteerDiff against multiple red-teaming strategies to assess its robustness. Finally, we explore the potential of SteerDiff for concept forgetting tasks, demonstrating its versatility in text-conditioned image generation.

Title: NETS: A Non-Equilibrium Transport Sampler

• Authors: Michael S. Albergo, Eric Vanden-Eijnden
• Subjects: cs.LG, cond-mat.stat-mech, hep-lat
• Abstract URL: https://arxiv.org/abs/2410.02711
• Pdf URL: https://arxiv.org/pdf/2410.02711
• Copy Paste: [[2410.02711]] NETS: A Non-Equilibrium Transport Sampler(https://arxiv.org/abs/2410.02711)
• Keywords: diffusion
• Abstract: We propose an algorithm, termed the Non-Equilibrium Transport Sampler (NETS), to sample from unnormalized probability distributions. NETS can be viewed as a variant of annealed importance sampling (AIS) based on Jarzynski's equality, in which the stochastic differential equation used to perform the non-equilibrium sampling is augmented with an additional learned drift term that lowers the impact of the unbiasing weights used in AIS. We show that this drift is the minimizer of a variety of objective functions, which can all be estimated in an unbiased fashion without backpropagating through solutions of the stochastic differential equations governing the sampling. We also prove that some these objectives control the Kullback-Leibler divergence of the estimated distribution from its target. NETS is shown to be unbiased and, in addition, has a tunable diffusion coefficient which can be adjusted post-training to maximize the effective sample size. We demonstrate the efficacy of the method on standard benchmarks, high-dimensional Gaussian mixture distributions, and a model from statistical lattice field theory, for which it surpasses the performances of related work and existing baselines.

Title: SynthFormer: Equivariant Pharmacophore-based Generation of Molecules for Ligand-Based Drug Design

• Authors: Zygimantas Jocys, Henriette M.G. Willems, Katayoun Farrahi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02718
• Pdf URL: https://arxiv.org/pdf/2410.02718
• Copy Paste: [[2410.02718]] SynthFormer: Equivariant Pharmacophore-based Generation of Molecules for Ligand-Based Drug Design(https://arxiv.org/abs/2410.02718)
• Keywords: generative
• Abstract: Drug discovery is a complex and resource-intensive process, with significant time and cost investments required to bring new medicines to patients. Recent advancements in generative machine learning (ML) methods offer promising avenues to accelerate early-stage drug discovery by efficiently exploring chemical space. This paper addresses the gap between in silico generative approaches and practical in vitro methodologies, highlighting the need for their integration to optimize molecule discovery. We introduce SynthFormer, a novel ML model that utilizes a 3D equivariant encoder for pharmacophores to generate fully synthesizable molecules, constructed as synthetic trees. Unlike previous methods, SynthFormer incorporates 3D information and provides synthetic paths, enhancing its ability to produce molecules with good docking scores across various proteins. Our contributions include a new methodology for efficient chemical space exploration using 3D information, a novel architecture called Synthformer for translating 3D pharmacophore representations into molecules, and a meaningful embedding space that organizes reagents for drug discovery optimization. Synthformer generates molecules that dock well and enables effective late-stage optimization restricted by synthesis paths.

Title: UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation

• Authors: Zixuan Li, Jing Xiong, Fanghua Ye, Chuanyang Zheng, Xun Wu, Jianqiao Lu, Zhongwei Wan, Xiaodan Liang, Chengming Li, Zhenan Sun, Lingpeng Kong, Ngai Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02719
• Pdf URL: https://arxiv.org/pdf/2410.02719
• Copy Paste: [[2410.02719]] UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation(https://arxiv.org/abs/2410.02719)
• Keywords: robust, large language model
• Abstract: We present UncertaintyRAG, a novel approach for long-context Retrieval-Augmented Generation (RAG) that utilizes Signal-to-Noise Ratio (SNR)-based span uncertainty to estimate similarity between text chunks. This span uncertainty enhances model calibration, improving robustness and mitigating semantic inconsistencies introduced by random chunking. Leveraging this insight, we propose an efficient unsupervised learning technique to train the retrieval model, alongside an effective data sampling and scaling strategy. UncertaintyRAG outperforms baselines by 2.03% on LLaMA-2-7B, achieving state-of-the-art results while using only 4% of the training data compared to other advanced open-source retrieval models under distribution shift settings. Our method demonstrates strong calibration through span uncertainty, leading to improved generalization and robustness in long-context RAG tasks. Additionally, UncertaintyRAG provides a lightweight retrieval model that can be integrated into any large language model with varying context window lengths, without the need for fine-tuning, showcasing the flexibility of our approach.

Title: Curvature Diversity-Driven Deformation and Domain Alignment for Point Cloud

• Authors: Mengxi Wu, Hao Huang, Yi Fang, Mohammad Rostami
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02720
• Pdf URL: https://arxiv.org/pdf/2410.02720
• Copy Paste: [[2410.02720]] Curvature Diversity-Driven Deformation and Domain Alignment for Point Cloud(https://arxiv.org/abs/2410.02720)
• Keywords: segmentation
• Abstract: Unsupervised Domain Adaptation (UDA) is crucial for reducing the need for extensive manual data annotation when training deep networks on point cloud data. A significant challenge of UDA lies in effectively bridging the domain gap. To tackle this challenge, we propose \textbf{C}urvature \textbf{D}iversity-Driven \textbf{N}uclear-Norm Wasserstein \textbf{D}omain Alignment (CDND). Our approach first introduces a \textit{\textbf{Curv}ature Diversity-driven Deformation \textbf{Rec}onstruction (CurvRec)} task, which effectively mitigates the gap between the source and target domains by enabling the model to extract salient features from semantically rich regions of a given point cloud. We then propose \textit{\textbf{D}eformation-based \textbf{N}uclear-norm \textbf{W}asserstein \textbf{D}iscrepancy (D-NWD)}, which applies the Nuclear-norm Wasserstein Discrepancy to both \textit{deformed and original} data samples to align the source and target domains. Furthermore, we contribute a theoretical justification for the effectiveness of D-NWD in distribution alignment and demonstrate that it is \textit{generic} enough to be applied to \textbf{any} deformations. To validate our method, we conduct extensive experiments on two public domain adaptation datasets for point cloud classification and segmentation tasks. Empirical experiment results show that our CDND achieves state-of-the-art performance by a noticeable margin over existing approaches.

Title: Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization

• Authors: Ryan C. Barron, Ves Grantcharov, Selma Wanna, Maksim E. Eren, Manish Bhattarai, Nicholas Solovyev, George Tompkins, Charles Nicholas, Kim Ø. Rasmussen, Cynthia Matuszek, Boian S. Alexandrov
• Subjects: cs.CL, cs.AI, cs.IR, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.02721
• Pdf URL: https://arxiv.org/pdf/2410.02721
• Copy Paste: [[2410.02721]] Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization(https://arxiv.org/abs/2410.02721)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are pre-trained on large-scale corpora and excel in numerous general natural language processing (NLP) tasks, such as question answering (QA). Despite their advanced language capabilities, when it comes to domain-specific and knowledge-intensive tasks, LLMs suffer from hallucinations, knowledge cut-offs, and lack of knowledge attributions. Additionally, fine tuning LLMs' intrinsic knowledge to highly specific domains is an expensive and time consuming process. The retrieval-augmented generation (RAG) process has recently emerged as a method capable of optimization of LLM responses, by referencing them to a predetermined ontology. It was shown that using a Knowledge Graph (KG) ontology for RAG improves the QA accuracy, by taking into account relevant sub-graphs that preserve the information in a structured manner. In this paper, we introduce SMART-SLIC, a highly domain-specific LLM framework, that integrates RAG with KG and a vector store (VS) that store factual domain specific information. Importantly, to avoid hallucinations in the KG, we build these highly domain-specific KGs and VSs without the use of LLMs, but via NLP, data mining, and nonnegative tensor factorization with automatic model selection. Pairing our RAG with a domain-specific: (i) KG (containing structured information), and (ii) VS (containing unstructured information) enables the development of domain-specific chat-bots that attribute the source of information, mitigate hallucinations, lessen the need for fine-tuning, and excel in highly domain-specific question answering tasks. We pair SMART-SLIC with chain-of-thought prompting agents. The framework is designed to be generalizable to adapt to any specific or specialized domain. In this paper, we demonstrate the question answering capabilities of our framework on a corpus of scientific publications on malware analysis and anomaly detection.

Title: Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation

• Authors: Rohin Manvi, Anikait Singh, Stefano Ermon
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02725
• Pdf URL: https://arxiv.org/pdf/2410.02725
• Copy Paste: [[2410.02725]] Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation(https://arxiv.org/abs/2410.02725)
• Keywords: generative, large language model
• Abstract: Inference-time computation is a powerful paradigm to enhance the performance of large language models (LLMs), with Best-of-N sampling being a widely used technique. However, this method is computationally expensive, requiring both (1) an external reward model and (2) the generation of multiple samples. In this work, we introduce a new generative self-evaluation scheme designed to adaptively reduce the number of generated samples while maintaining or even improving performance. We use a generative reward model formulation, allowing the LLM to predict mid-generation the probability that restarting the generation will yield a better response. These predictions are obtained without an external reward model and can be used to decide whether or not to generate more samples, prune unpromising samples early on, or to pick the best sample. This capability is very inexpensive as it involves generating a single predefined token. Trained using a dataset constructed with real unfiltered LMSYS user prompts, Llama 3.1 8B's win rate against GPT-4 on AlpacaEval increases from 21% to 34% with 16 samples and math performance on GSM8K improves from 84% to 91%. By sampling only when the LLM determines that it is beneficial to do so and adaptively adjusting temperature annealing, we demonstrate that 74% of the improvement from using 16 samples can be achieved with only 1.2 samples on average. We further demonstrate that 50-75% of samples can be pruned early in generation with minimal degradation in performance. Overall, our methods enable more efficient and scalable compute utilization during inference for LLMs.

Title: Data Similarity-Based One-Shot Clustering for Multi-Task Hierarchical Federated Learning

• Authors: Abdulmoneam Ali, Ahmed Arafa
• Subjects: cs.LG, cs.IT, cs.NI, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.02733
• Pdf URL: https://arxiv.org/pdf/2410.02733
• Copy Paste: [[2410.02733]] Data Similarity-Based One-Shot Clustering for Multi-Task Hierarchical Federated Learning(https://arxiv.org/abs/2410.02733)
• Keywords: privacy, extraction, federate
• Abstract: We address the problem of cluster identity estimation in a hierarchical federated learning setting in which users work toward learning different tasks. To overcome the challenge of task heterogeneity, users need to be grouped in a way such that users with the same task are in the same group, conducting training together, while sharing the weights of feature extraction layers with the other groups. Toward that end, we propose a one-shot clustering algorithm that can effectively identify and group users based on their data similarity. This enables more efficient collaboration and sharing of a common layer representation within the federated learning system. Our proposed algorithm not only enhances the clustering process, but also overcomes challenges related to privacy concerns, communication overhead, and the need for prior knowledge about learning models or loss function behaviors. We validate our proposed algorithm using various datasets such as CIFAR-10 and Fashion MNIST, and show that it outperforms the baseline in terms of accuracy and variance reduction.

Title: Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models

• Authors: Zhengfeng Lai, Vasileios Saveris, Chen Chen, Hong-You Chen, Haotian Zhang, Bowen Zhang, Juan Lao Tebar, Wenze Hu, Zhe Gan, Peter Grasch, Meng Cao, Yinfei Yang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02740
• Pdf URL: https://arxiv.org/pdf/2410.02740
• Copy Paste: [[2410.02740]] Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models(https://arxiv.org/abs/2410.02740)
• Keywords: diffusion
• Abstract: Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models.

Title: Salient Information Prompting to Steer Content in Prompt-based Abstractive Summarization

• Authors: Lei Xu, Mohammed Asad Karim, Saket Dingliwal, Aparna Elangovan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02741
• Pdf URL: https://arxiv.org/pdf/2410.02741
• Copy Paste: [[2410.02741]] Salient Information Prompting to Steer Content in Prompt-based Abstractive Summarization(https://arxiv.org/abs/2410.02741)
• Keywords: large language model
• Abstract: Large language models (LLMs) can generate fluent summaries across domains using prompting techniques, reducing the need to train models for summarization applications. However, crafting effective prompts that guide LLMs to generate summaries with the appropriate level of detail and writing style remains a challenge. In this paper, we explore the use of salient information extracted from the source document to enhance summarization prompts. We show that adding keyphrases in prompts can improve ROUGE F1 and recall, making the generated summaries more similar to the reference and more complete. The number of keyphrases can control the precision-recall trade-off. Furthermore, our analysis reveals that incorporating phrase-level salient information is superior to word- or sentence-level. However, the impact on hallucination is not universally positive across LLMs. To conduct this analysis, we introduce Keyphrase Signal Extractor (SigExt), a lightweight model that can be finetuned to extract salient keyphrases. By using SigExt, we achieve consistent ROUGE improvements across datasets and open-weight and proprietary LLMs without any LLM customization. Our findings provide insights into leveraging salient information in building prompt-based summarization systems.

Title: Grounding Large Language Models In Embodied Environment With Imperfect World Models

• Authors: Haolan Liu, Jishen Zhao
• Subjects: cs.CL, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.02742
• Pdf URL: https://arxiv.org/pdf/2410.02742
• Copy Paste: [[2410.02742]] Grounding Large Language Models In Embodied Environment With Imperfect World Models(https://arxiv.org/abs/2410.02742)
• Keywords: large language model
• Abstract: Despite a widespread success in various applications, large language models (LLMs) often stumble when tackling basic physical reasoning or executing robotics tasks, due to a lack of direct experience with the physical nuances of the real world. To address these issues, we propose a Grounding Large language model with Imperfect world MOdel (GLIMO), which utilizes proxy world models such as simulators to collect and synthesize trining data. GLIMO incorporates an LLM agent-based data generator to automatically create high-quality and diverse instruction datasets. The generator includes an iterative self-refining module for temporally consistent experience sampling, a diverse set of question-answering instruction seeds, and a retrieval-augmented generation module for reflecting on prior experiences. Comprehensive experiments show that our approach improve the performance of strong open-source LLMs like LLaMA-3 with a performance boost of 2.04 $\times$, 1.54 $\times$, and 1.82 $\times$ across three different benchmarks, respectively. The performance is able to compete with or surpass their larger counterparts such as GPT-4.

Title: MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions

• Authors: Yekun Chai, Haoran Sun, Huang Fang, Shuohuan Wang, Yu Sun, Hua Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02743
• Pdf URL: https://arxiv.org/pdf/2410.02743
• Copy Paste: [[2410.02743]] MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions(https://arxiv.org/abs/2410.02743)
• Keywords: large language model
• Abstract: Reinforcement learning from human feedback (RLHF) has demonstrated effectiveness in aligning large language models (LLMs) with human preferences. However, token-level RLHF suffers from the credit assignment problem over long sequences, where delayed rewards make it challenging for the model to discern which actions contributed to successful outcomes. This hinders learning efficiency and slows convergence. In this paper, we propose MA-RLHF, a simple yet effective RLHF framework that incorporates macro actions -- sequences of tokens or higher-level language constructs -- into the learning process. By operating at this higher level of abstraction, our approach reduces the temporal distance between actions and rewards, facilitating faster and more accurate credit assignment. This results in more stable policy gradient estimates and enhances learning efficiency within each episode, all without increasing computational complexity during training or inference. We validate our approach through extensive experiments across various model sizes and tasks, including text summarization, dialogue generation, question answering, and program synthesis. Our method achieves substantial performance improvements over standard RLHF, with performance gains of up to 30% in text summarization and code generation, 18% in dialogue, and 8% in question answering tasks. Notably, our approach reaches parity with vanilla RLHF 1.7x to 2x faster in terms of training time and continues to outperform it with further training. We will make our code and data publicly available at this https URL .

Title: Neutral residues: revisiting adapters for model extension

• Authors: Franck Signe Talla, Herve Jegou, Edouard Grave
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02744
• Pdf URL: https://arxiv.org/pdf/2410.02744
• Copy Paste: [[2410.02744]] Neutral residues: revisiting adapters for model extension(https://arxiv.org/abs/2410.02744)
• Keywords: large language model
• Abstract: We address the problem of extending a pretrained large language model to a new domain that was not seen at training time, like adding a language for which the original model has seen no or little training data. Popular solutions like fine-tuning or low-rank adaptation are successful at domain adaptation, but formally they do not add any extra capacity and degrade the performance in the original domain. Our paper analyzes this extension problem under three angles: data, architecture and training procedure, which are advantageously considered jointly. In particular, we improve adapters and make it possible to learn an entire new language while ensuring that the output of the neural network is almost unchanged in the original domain. For this purpose, we modify the new residual blocks in a way that leads each new residual block to output near-zeros in the original domain. This solution of neutral residues, which borrows architectural components from mixture of experts, is effective: with only 20% extra learnable weights compared to an original model trained on English, we get results that are significantly better than concurrent approaches (fine-tuning, low-rank or vanilla adapters) in terms of the trade-off between learning a new language and not forgetting English.

Title: AVG-LLaVA: A Multimodal Large Model with Adaptive Visual Granularity

• Authors: Zhibin Lan, Liqiang Niu, Fandong Meng, Wenbo Li, Jie Zhou, Jinsong Su
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02745
• Pdf URL: https://arxiv.org/pdf/2410.02745
• Copy Paste: [[2410.02745]] AVG-LLaVA: A Multimodal Large Model with Adaptive Visual Granularity(https://arxiv.org/abs/2410.02745)
• Keywords: transformer
• Abstract: Recently, when dealing with high-resolution images, dominant LMMs usually divide them into multiple local images and one global image, which will lead to a large number of visual tokens. In this work, we introduce AVG-LLaVA, an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction. This approach not only reduces the number of visual tokens and speeds up inference, but also improves the overall model performance. Specifically, we introduce the following modules based on LLaVA-NeXT: (a) a visual granularity scaler that includes multiple pooling layers to obtain visual tokens with different granularities; (b) a visual granularity router, which includes a Transformer layer, an MLP layer, and a voter layer, used to select the appropriate visual granularity based on the image and instruction. Furthermore, we propose RGLF, a novel training paradigm that aims at aligning the granularity predicted by the router with the preferences of the LMM, without the need for additional manually annotated data. Extensive experiments and analysis show that AVG-LLaVA achieves superior performance across 11 benchmarks, as well as significantly reduces the number of visual tokens and speeds up inference (e.g., an 85.3% reduction in visual tokens and a 2.53$\times$ increase in inference speed on the AI2D benchmark).

Title: Contrastive Localized Language-Image Pre-Training

• Authors: Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, Zhe Gan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02746
• Pdf URL: https://arxiv.org/pdf/2410.02746
• Copy Paste: [[2410.02746]] Contrastive Localized Language-Image Pre-Training(https://arxiv.org/abs/2410.02746)
• Keywords: large language model
• Abstract: Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, CLIP has been widely adopted as the vision backbone of multimodal large language models (MLLMs) to connect image inputs for language interactions. The success of CLIP as a vision-language foundation model relies on aligning web-crawled noisy text annotations at image levels. Nevertheless, such criteria may become insufficient for downstream tasks in need of fine-grained vision representations, especially when region-level understanding is demanding for MLLMs. In this paper, we improve the localization capability of CLIP with several advances. We propose a pre-training method called Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings, of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text pseudo-labels at scale. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.

Title: CriSPO: Multi-Aspect Critique-Suggestion-guided Automatic Prompt Optimization for Text Generation

• Authors: Han He, Qianchu Liu, Lei Xu, Chaitanya Shivade, Yi Zhang, Sundararajan Srinivasan, Katrin Kirchhoff
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02748
• Pdf URL: https://arxiv.org/pdf/2410.02748
• Copy Paste: [[2410.02748]] CriSPO: Multi-Aspect Critique-Suggestion-guided Automatic Prompt Optimization for Text Generation(https://arxiv.org/abs/2410.02748)
• Keywords: large language model
• Abstract: Large language models (LLMs) can generate fluent summaries across domains using prompting techniques, reducing the need to train models for summarization applications. However, crafting effective prompts that guide LLMs to generate summaries with the appropriate level of detail and writing style remains a challenge. In this paper, we explore the use of salient information extracted from the source document to enhance summarization prompts. We show that adding keyphrases in prompts can improve ROUGE F1 and recall, making the generated summaries more similar to the reference and more complete. The number of keyphrases can control the precision-recall trade-off. Furthermore, our analysis reveals that incorporating phrase-level salient information is superior to word- or sentence-level. However, the impact on hallucination is not universally positive across LLMs. To conduct this analysis, we introduce Keyphrase Signal Extractor (CriSPO), a lightweight model that can be finetuned to extract salient keyphrases. By using CriSPO, we achieve consistent ROUGE improvements across datasets and open-weight and proprietary LLMs without any LLM customization. Our findings provide insights into leveraging salient information in building prompt-based summarization systems.

Title: Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

• Authors: Ulyana Piterbarg, Lerrel Pinto, Rob Fergus
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02749
• Pdf URL: https://arxiv.org/pdf/2410.02749
• Copy Paste: [[2410.02749]] Training Language Models on Synthetic Edit Sequences Improves Code Synthesis(https://arxiv.org/abs/2410.02749)
• Keywords: large language model
• Abstract: Software engineers mainly write code by editing existing programs. In contrast, large language models (LLMs) autoregressively synthesize programs in a single pass. One explanation for this is the scarcity of open-sourced edit data. While high-quality instruction data for code synthesis is already scarce, high-quality edit data is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors existing code into a sequence of code edits by using a linter to procedurally sample across the error-free insertions that can be used to sequentially write programs. It outputs edit sequences as text strings consisting of consecutive program diffs. To test LintSeq, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-sequence tuples. Then, we instruction finetune a series of smaller LLMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset, comparing zero-shot performance on code synthesis benchmarks. We show that during repeated sampling, edit sequence finetuned models produce more diverse programs than baselines. This results in better inference-time scaling for benchmark coverage as a function of samples, i.e. the fraction of problems "pass@k" solved by any attempt given "k" tries. For example, on HumanEval pass@50, small LLMs finetuned on synthetic edit sequences are competitive with GPT-4 and outperform models finetuned on the baseline dataset by +20% (+/-3%) in absolute score. Finally, we also pretrain our own tiny LMs for code understanding. We show that finetuning tiny models on synthetic code edits results in state-of-the-art code synthesis for the on-device model class. Our 150M parameter edit sequence LM matches or outperforms code models with twice as many parameters, both with and without repeated sampling, including Codex and AlphaCode.

Title: SIEVE: General Purpose Data Filtering System Matching GPT-4o Accuracy at 1% the Cost

• Authors: Jifan Zhang, Robert Nowak
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02755
• Pdf URL: https://arxiv.org/pdf/2410.02755
• Copy Paste: [[2410.02755]] SIEVE: General Purpose Data Filtering System Matching GPT-4o Accuracy at 1% the Cost(https://arxiv.org/abs/2410.02755)
• Keywords: large language model
• Abstract: Creating specialized large language models requires vast amounts of clean, special purpose data for training and fine-tuning. With only a handful of existing large-scale, domain-specific datasets, creation of new datasets is required in most applications. This requires the development of new application-specific filtering of web-scale data. Filtering with a high-performance, general-purpose LLM such as GPT-4o can be highly effective, but this is extremely expensive at web-scale. This paper proposes SIEVE, a lightweight alternative that matches GPT-4o accuracy at a fraction of the cost. SIEVE can perform up to 500 filtering operations for the cost of one GPT-4o filtering call. The key to SIEVE is a seamless integration of GPT-4o and lightweight T5 models, using active learning to fine-tune T5 in the background with a small number of calls to GPT-4o. Once trained, it performs as well as GPT-4o at a tiny fraction of the cost. We experimentally validate SIEVE on the OpenWebText dataset, using five highly customized filter tasks targeting high quality and domain-specific content. Our results demonstrate the effectiveness and efficiency of our method in curating large, high-quality datasets for language model training at a substantially lower cost (1%) than existing techniques. To further validate SIEVE, experiments show that SIEVE and GPT-4o achieve similar accuracy, with human evaluators preferring SIEVE's filtering results to those of GPT-4o.

Title: Loong: Generating Minute-level Long Videos with Autoregressive Language Models

• Authors: Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, Xihui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02757
• Pdf URL: https://arxiv.org/pdf/2410.02757
• Copy Paste: [[2410.02757]] Loong: Generating Minute-level Long Videos with Autoregressive Language Models(https://arxiv.org/abs/2410.02757)
• Keywords: large language model
• Abstract: It is desirable but challenging to generate content-rich long videos in the scale of minutes. Autoregressive large language models (LLMs) have achieved great success in generating coherent and long sequences of tokens in the domain of natural language processing, while the exploration of autoregressive LLMs for video generation is limited to generating short videos of several seconds. In this work, we conduct a deep analysis of the challenges that prevent autoregressive LLM-based video generators from generating long videos. Based on the observations and analysis, we propose Loong, a new autoregressive LLM-based video generator that can generate minute-long videos. Specifically, we model the text tokens and video tokens as a unified sequence for autoregressive LLMs and train the model from scratch. We propose progressive short-to-long training with a loss re-weighting scheme to mitigate the loss imbalance problem for long video training. We further investigate inference strategies, including video token re-encoding and sampling strategies, to diminish error accumulation during inference. Our proposed Loong can be trained on 10-second videos and be extended to generate minute-level long videos conditioned on text prompts, as demonstrated by the results. More samples are available at: this https URL.

Title: Erasing Conceptual Knowledge from Language Models

• Authors: Rohit Gandikota, Sheridan Feucht, Samuel Marks, David Bau
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02760
• Pdf URL: https://arxiv.org/pdf/2410.02760
• Copy Paste: [[2410.02760]] Erasing Conceptual Knowledge from Language Models(https://arxiv.org/abs/2410.02760)
• Keywords: security, attack, robust
• Abstract: Concept erasure in language models has traditionally lacked a comprehensive evaluation framework, leading to incomplete assessments of effectiveness of erasure methods. We propose an evaluation paradigm centered on three critical criteria: innocence (complete knowledge removal), seamlessness (maintaining conditional fluent generation), and specificity (preserving unrelated task performance). Our evaluation metrics naturally motivate the development of Erasure of Language Memory (ELM), a new method designed to address all three dimensions. ELM employs targeted low-rank updates to alter output distributions for erased concepts while preserving overall model capabilities including fluency when prompted for an erased concept. We demonstrate ELM's efficacy on biosecurity, cybersecurity, and literary domain erasure tasks. Comparative analysis shows that ELM achieves superior performance across our proposed metrics, including near-random scores on erased topic assessments, generation fluency, maintained accuracy on unrelated benchmarks, and robustness under adversarial attacks. Our code, data, and trained models are available at this https URL

Title: FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models

• Authors: Zhipei Xu, Xuanyu Zhang, Runyi Li, Zecheng Tang, Qing Huang, Jian Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02761
• Pdf URL: https://arxiv.org/pdf/2410.02761
• Copy Paste: [[2410.02761]] FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models(https://arxiv.org/abs/2410.02761)
• Keywords: generative, large language model
• Abstract: The rapid development of generative AI is a double-edged sword, which not only facilitates content creation but also makes image manipulation easier and more difficult to detect. Although current image forgery detection and localization (IFDL) methods are generally effective, they tend to face two challenges: \textbf{1)} black-box nature with unknown detection principle, \textbf{2)} limited generalization across diverse tampering methods (e.g., Photoshop, DeepFake, AIGC-Editing). To address these issues, we propose the explainable IFDL task and design FakeShield, a multi-modal framework capable of evaluating image authenticity, generating tampered region masks, and providing a judgment basis based on pixel-level and image-level tampering clues. Additionally, we leverage GPT-4o to enhance existing IFDL datasets, creating the Multi-Modal Tamper Description dataSet (MMTD-Set) for training FakeShield's tampering analysis capabilities. Meanwhile, we incorporate a Domain Tag-guided Explainable Forgery Detection Module (DTE-FDM) and a Multi-modal Forgery Localization Module (MFLM) to address various types of tamper detection interpretation and achieve forgery localization guided by detailed textual descriptions. Extensive experiments demonstrate that FakeShield effectively detects and localizes various tampering techniques, offering an explainable and superior solution compared to previous IFDL methods.

Title: Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations

• Authors: Nick Jiang, Anish Kachinthaya, Suzie Petryk, Yossi Gandelsman
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02762
• Pdf URL: https://arxiv.org/pdf/2410.02762
• Copy Paste: [[2410.02762]] Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations(https://arxiv.org/abs/2410.02762)
• Keywords: segmentation
• Abstract: We investigate the internal representations of vision-language models (VLMs) to address hallucinations, a persistent challenge despite advances in model size and training. We project VLMs' internal image representations to their language vocabulary and observe more confident output probabilities on real objects than hallucinated objects. We additionally use these output probabilities to spatially localize real objects. Building on this approach, we introduce a knowledge erasure algorithm that removes hallucinations by linearly orthogonalizing image features with respect to hallucinated object features. We show that targeted edits to a model's latent representations can reduce hallucinations by up to 25.7% on the COCO2014 dataset while preserving performance. Our findings demonstrate how a deeper understanding of VLMs' latent representations can enhance reliability and enable novel capabilities, such as zero-shot segmentation.