2024-10-03

Title: Text Clustering as Classification with LLMs

• Authors: Chen Huang, Guoxiu He
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.00927
• Pdf URL: https://arxiv.org/pdf/2410.00927
• Copy Paste: [[2410.00927]] Text Clustering as Classification with LLMs(https://arxiv.org/abs/2410.00927)
• Keywords: large language model
• Abstract: Text clustering remains valuable in real-world applications where manual labeling is cost-prohibitive. It facilitates efficient organization and analysis of information by grouping similar texts based on their representations. However, implementing this approach necessitates fine-tuned embedders for downstream data and sophisticated similarity metrics. To address this issue, this study presents a novel framework for text clustering that effectively leverages the in-context learning capacity of Large Language Models (LLMs). Instead of fine-tuning embedders, we propose to transform the text clustering into a classification task via LLM. First, we prompt LLM to generate potential labels for a given dataset. Second, after integrating similar labels generated by the LLM, we prompt the LLM to assign the most appropriate label to each sample in the dataset. Our framework has been experimentally proven to achieve comparable or superior performance to state-of-the-art clustering methods that employ embeddings, without requiring complex fine-tuning or clustering algorithms. We make our code available to the public for utilization at this https URL.

Title: ACEV: Unsupervised Intersecting Manifold Segmentation using Adaptation to Angular Change of Eigenvectors in Intrinsic Dimension

• Authors: Subhadip Boral, Rikathi Pal, Ashish Ghosh
• Subjects: cs.LG, cs.AI, cs.CG
• Abstract URL: https://arxiv.org/abs/2410.00930
• Pdf URL: https://arxiv.org/pdf/2410.00930
• Copy Paste: [[2410.00930]] ACEV: Unsupervised Intersecting Manifold Segmentation using Adaptation to Angular Change of Eigenvectors in Intrinsic Dimension(https://arxiv.org/abs/2410.00930)
• Keywords: segmentation
• Abstract: Intersecting manifold segmentation has been a focus of research, where individual manifolds, that intersect with other manifolds, are separated to discover their distinct properties. The proposed method is based on the intuition that when a manifold in $D$ dimensional space with an intrinsic dimension of $d$ intersects with another manifold, the data variance grows in more than $d$ directions. The proposed method measures local data variances and determines their vector directions. It counts the number of vectors with non-zero variance, which determines the manifold's intrinsic dimension. For detection of the intersection region, the method adapts to the changes in the angular gaps between the corresponding direction vectors of the child and parent using exponential moving averages using a tree structure construction. Accordingly, it includes those data points in the same manifold whose neighborhood is within the adaptive angular difference and eventually identifies the data points in the intersection area of manifolds. Data points whose inclusion in the neighborhood-identified data points increases their intrinsic dimensionality are removed based on data variance and distance. The proposed method performs better than 18 SOTA manifold segmentation methods in ARI and NMI scores over 14 real-world datasets with lesser time complexity and better stability.

Title: MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

• Authors: Sheng Wang, Liheng Chen, Pengan Chen, Jingwei Dong, Boyang Xue, Jiyue Jiang, Lingpeng Kong, Chuan Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.00938
• Pdf URL: https://arxiv.org/pdf/2410.00938
• Copy Paste: [[2410.00938]] MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards(https://arxiv.org/abs/2410.00938)
• Keywords: large language model
• Abstract: The rapid scaling of large language models necessitates more lightweight finetuning methods to reduce the explosive GPU memory overhead when numerous customized models are served simultaneously. Targeting more parameter-efficient low-rank adaptation (LoRA), parameter sharing presents a promising solution. Empirically, our research into high-level sharing principles highlights the indispensable role of differentiation in reversing the detrimental effects of pure sharing. Guided by this finding, we propose Mixture of Shards (MoS), incorporating both inter-layer and intra-layer sharing schemes, and integrating four nearly cost-free differentiation strategies, namely subset selection, pair dissociation, vector sharding, and shard privatization. Briefly, it selects a designated number of shards from global pools with a Mixture-of-Experts (MoE)-like routing mechanism before sequentially concatenating them to low-rank matrices. Hence, it retains all the advantages of LoRA while offering enhanced parameter efficiency, and effectively circumvents the drawbacks of peer parameter-sharing methods. Our empirical experiments demonstrate approximately 8x parameter savings in a standard LoRA setting. The ablation study confirms the significance of each component. Our insights into parameter sharing and MoS method may illuminate future developments of more parameter-efficient finetuning methods.

Title: RisingBALLER: A player is a token, a match is a sentence, A path towards a foundational model for football players data analytics

• Authors: Akedjou Achraff Adjileye
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.00943
• Pdf URL: https://arxiv.org/pdf/2410.00943
• Copy Paste: [[2410.00943]] RisingBALLER: A player is a token, a match is a sentence, A path towards a foundational model for football players data analytics(https://arxiv.org/abs/2410.00943)
• Keywords: transformer
• Abstract: In this paper, I introduce RisingBALLER, the first publicly available approach that leverages a transformer model trained on football match data to learn match-specific player representations. Drawing inspiration from advances in language modeling, RisingBALLER treats each football match as a unique sequence in which players serve as tokens, with their embeddings shaped by the specific context of the match. Through the use of masked player prediction (MPP) as a pre-training task, RisingBALLER learns foundational features for football player representations, similar to how language models learn semantic features for text representations. As a downstream task, I introduce next match statistics prediction (NMSP) to showcase the effectiveness of the learned player embeddings. The NMSP model surpasses a strong baseline commonly used for performance forecasting within the community. Furthermore, I conduct an in-depth analysis to demonstrate how the learned embeddings by RisingBALLER can be used in various football analytics tasks, such as producing meaningful positional features that capture the essence and variety of player roles beyond rigid x,y coordinates, team cohesion estimation, and similar player retrieval for more effective data-driven scouting. More than a simple machine learning model, RisingBALLER is a comprehensive framework designed to transform football data analytics by learning high-level foundational features for players, taking into account the context of each match. It offers a deeper understanding of football players beyond individual statistics.

Title: Robust Guided Diffusion for Offline Black-Box Optimization

• Authors: Can (Sam)Chen, Christopher Beckham, Zixuan Liu, Xue Liu, Christopher Pal
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.00983
• Pdf URL: https://arxiv.org/pdf/2410.00983
• Copy Paste: [[2410.00983]] Robust Guided Diffusion for Offline Black-Box Optimization(https://arxiv.org/abs/2410.00983)
• Keywords: robust, diffusion
• Abstract: Offline black-box optimization aims to maximize a black-box function using an offline dataset of designs and their measured properties. Two main approaches have emerged: the forward approach, which learns a mapping from input to its value, thereby acting as a proxy to guide optimization, and the inverse approach, which learns a mapping from value to input for conditional generation. (a) Although proxy-free~(classifier-free) diffusion shows promise in robustly modeling the inverse mapping, it lacks explicit guidance from proxies, essential for generating high-performance samples beyond the training distribution. Therefore, we propose \textit{proxy-enhanced sampling} which utilizes the explicit guidance from a trained proxy to bolster proxy-free diffusion with enhanced sampling control. (b) Yet, the trained proxy is susceptible to out-of-distribution issues. To address this, we devise the module \textit{diffusion-based proxy refinement}, which seamlessly integrates insights from proxy-free diffusion back into the proxy for refinement. To sum up, we propose \textit{\textbf{R}obust \textbf{G}uided \textbf{D}iffusion for Offline Black-box Optimization}~(\textbf{RGD}), combining the advantages of proxy~(explicit guidance) and proxy-free diffusion~(robustness) for effective conditional generation. RGD achieves state-of-the-art results on various design-bench tasks, underscoring its efficacy. Our code is at this https URL.

Title: Tackling the Accuracy-Interpretability Trade-off in a Hierarchy of Machine Learning Models for the Prediction of Extreme Heatwaves

• Authors: Alessandro Lovo, Amaury Lancelin, Corentin Herbert, Freddy Bouchet
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2410.00984
• Pdf URL: https://arxiv.org/pdf/2410.00984
• Copy Paste: [[2410.00984]] Tackling the Accuracy-Interpretability Trade-off in a Hierarchy of Machine Learning Models for the Prediction of Extreme Heatwaves(https://arxiv.org/abs/2410.00984)
• Keywords: interpretability
• Abstract: When performing predictions that use Machine Learning (ML), we are mainly interested in performance and interpretability. This generates a natural trade-off, where complex models generally have higher skills but are harder to explain and thus trust. Interpretability is particularly important in the climate community, where we aim at gaining a physical understanding of the underlying phenomena. Even more so when the prediction concerns extreme weather events with high impact on society. In this paper, we perform probabilistic forecasts of extreme heatwaves over France, using a hierarchy of increasingly complex ML models, which allows us to find the best compromise between accuracy and interpretability. More precisely, we use models that range from a global Gaussian Approximation (GA) to deep Convolutional Neural Networks (CNNs), with the intermediate steps of a simple Intrinsically Interpretable Neural Network (IINN) and a model using the Scattering Transform (ScatNet). Our findings reveal that CNNs provide higher accuracy, but their black-box nature severely limits interpretability, even when using state-of-the-art Explainable Artificial Intelligence (XAI) tools. In contrast, ScatNet achieves similar performance to CNNs while providing greater transparency, identifying key scales and patterns in the data that drive predictions. This study underscores the potential of interpretability in ML models for climate science, demonstrating that simpler models can rival the performance of their more complex counterparts, all the while being much easier to understand. This gained interpretability is crucial for building trust in model predictions and uncovering new scientific insights, ultimately advancing our understanding and management of extreme weather events.

Title: LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details

• Authors: Jian Yang, Xukun Wang, Wentao Wang, Guoming Li, Qihang Fang, Ruihong Yuan, Tianyang Wang, Jason Zhaoxin Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.00990
• Pdf URL: https://arxiv.org/pdf/2410.00990
• Copy Paste: [[2410.00990]] LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details(https://arxiv.org/abs/2410.00990)
• Keywords: robust
• Abstract: Audio-driven talking head generation is a pivotal area within film-making and Virtual Reality. Although existing methods have made significant strides following the end-to-end paradigm, they still encounter challenges in producing videos with high-frequency details due to their limited expressivity in this domain. This limitation has prompted us to explore an effective post-processing approach to synthesize photo-realistic talking head videos. Specifically, we employ a pretrained Wav2Lip model as our foundation model, leveraging its robust audio-lip alignment capabilities. Drawing on the theory of Lipschitz Continuity, we have theoretically established the noise robustness of Vector Quantised Auto Encoders (VQAEs). Our experiments further demonstrate that the high-frequency texture deficiency of the foundation model can be temporally consistently recovered by the Space-Optimised Vector Quantised Auto Encoder (SOVQAE) we introduced, thereby facilitating the creation of realistic talking head videos. We conduct experiments on both the conventional dataset and the High-Frequency TalKing head (HFTK) dataset that we curated. The results indicate that our method, LaDTalk, achieves new state-of-the-art video quality and out-of-domain lip synchronization performance.

Title: Y-CA-Net: A Convolutional Attention Based Network for Volumetric Medical Image Segmentation

• Authors: Muhammad Hamza Sharif, Muzammal Naseer, Mohammad Yaqub, Min Xu, Mohsen Guizani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01003
• Pdf URL: https://arxiv.org/pdf/2410.01003
• Copy Paste: [[2410.01003]] Y-CA-Net: A Convolutional Attention Based Network for Volumetric Medical Image Segmentation(https://arxiv.org/abs/2410.01003)
• Keywords: transformer, segmentation
• Abstract: Recent attention-based volumetric segmentation (VS) methods have achieved remarkable performance in the medical domain which focuses on modeling long-range dependencies. However, for voxel-wise prediction tasks, discriminative local features are key components for the performance of the VS models which is missing in attention-based VS methods. Aiming at resolving this issue, we deliberately incorporate the convolutional encoder branch with transformer backbone to extract local and global features in a parallel manner and aggregate them in Cross Feature Mixer Module (CFMM) for better prediction of segmentation mask. Consequently, we observe that the derived model, Y-CT-Net, achieves competitive performance on multiple medical segmentation tasks. For example, on multi-organ segmentation, Y-CT-Net achieves an 82.4% dice score, surpassing well-tuned VS Transformer/CNN-like baselines UNETR/ResNet-3D by 2.9%/1.4%. With the success of Y-CT-Net, we extend this concept with hybrid attention models, that derived Y-CH-Net model, which brings a 3% improvement in terms of HD95 score for same segmentation task. The effectiveness of both models Y-CT-Net and Y-CH-Net verifies our hypothesis and motivates us to initiate the concept of Y-CA-Net, a versatile generic architecture based upon any two encoders and a decoder backbones, to fully exploit the complementary strengths of both convolution and attention mechanisms. Based on experimental results, we argue Y-CA-Net is a key player in achieving superior results for volumetric segmentation.

Title: Machine Learning-Assisted Intrusion Detection for Enhancing Internet of Things Security

• Authors: Mona Esmaeili, Morteza Rahimi, Matin Khajavi, Dorsa Farahmand, Hadi Jabbari Saray
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01016
• Pdf URL: https://arxiv.org/pdf/2410.01016
• Copy Paste: [[2410.01016]] Machine Learning-Assisted Intrusion Detection for Enhancing Internet of Things Security(https://arxiv.org/abs/2410.01016)
• Keywords: secure, security, privacy, attack
• Abstract: Attacks against the Internet of Things (IoT) are rising as devices, applications, and interactions become more networked and integrated. The increase in cyber-attacks that target IoT networks poses a huge vulnerability and threat to the privacy, security, functionality, and availability of critical systems, which leads to operational disruptions, financial losses, identity thefts, and data breaches. To efficiently secure IoT devices, real-time detection of intrusion systems is critical, especially those using machine learning to identify threats and mitigate risks and vulnerabilities. This paper investigates the latest research on machine learning-based intrusion detection strategies for IoT security, concentrating on real-time responsiveness, detection accuracy, and algorithm efficiency. Key studies were reviewed from all well-known academic databases, and a taxonomy was provided for the existing approaches. This review also highlights existing research gaps and outlines the limitations of current IoT security frameworks to offer practical insights for future research directions and developments.

Title: A Generalized Approach to Root-based Attacks against PLWE

• Authors: Iván Blanco Chacón, Raúl Durán Díaz, Rodrigo Martín Sánchez-Ledesma
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.01017
• Pdf URL: https://arxiv.org/pdf/2410.01017
• Copy Paste: [[2410.01017]] A Generalized Approach to Root-based Attacks against PLWE(https://arxiv.org/abs/2410.01017)
• Keywords: attack, robust
• Abstract: The Polynomial Learning With Errors problem (PLWE) serves as the background of two of the three cryptosystems standardized in August 2024 by the National Institute of Standards and Technology to replace non-quantum resistant current primitives like those based on RSA, Diffie-Hellman or its elliptic curve analogue. Although PLWE is highly believed to be quantum resistant, this fact has not yet been established, contrariwise to other post-quantum proposals like multivariate and some code based ones. Moreover, several vulnerabilities have been encountered for a number of specific instances. In a search for more flexibility, it becomes fully relevant to study the robustness of PLWE based on other polynomials, not necessarily cyclotomic. In 2015, Elias et al found a good number of attacks based on different features of the roots of the polynomial. In the present work we present an overview of the approximations made against PLWE derived from this and subsequent works, along with several new attacks which refine those by Elias et al. exploiting the order of the trace of roots over finite extensions of the finite field under the three scenarios laid out by Elias et al., allowing to generalize the setting in which the attacks can be carried out.

Title: Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity

• Authors: Michael R. Metel, Peng Lu, Boxing Chen, Mehdi Rezagholizadeh, Ivan Kobyzev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01028
• Pdf URL: https://arxiv.org/pdf/2410.01028
• Copy Paste: [[2410.01028]] Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity(https://arxiv.org/abs/2410.01028)
• Keywords: large language model
• Abstract: We present a simple on the fly method for faster inference of large language models. Unlike other (self-)speculative decoding techniques, our method does not require fine-tuning or black-box optimization to generate a fixed draft model, relying instead on simple rules to generate varying draft models adapted to the input context. We show empirically that our light-weight algorithm is competitive with the current SOTA for self-speculative decoding, while being a truly plug-and-play method.

Title: FCE-YOLOv8: YOLOv8 with Feature Context Excitation Modules for Fracture Detection in Pediatric Wrist X-ray Images

• Authors: Rui-Yang Ju, Chun-Tse Chien, Enkaer Xieerke, Jen-Shiun Chiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01031
• Pdf URL: https://arxiv.org/pdf/2410.01031
• Copy Paste: [[2410.01031]] FCE-YOLOv8: YOLOv8 with Feature Context Excitation Modules for Fracture Detection in Pediatric Wrist X-ray Images(https://arxiv.org/abs/2410.01031)
• Keywords: transformer
• Abstract: Children often suffer wrist trauma in daily life, while they usually need radiologists to analyze and interpret X-ray images before surgical treatment by surgeons. The development of deep learning has enabled neural networks to serve as computer-assisted diagnosis (CAD) tools to help doctors and experts in medical image diagnostics. Since the You Only Look Once Version-8 (YOLOv8) model has obtained the satisfactory success in object detection tasks, it has been applied to various fracture detection. This work introduces four variants of Feature Contexts Excitation-YOLOv8 (FCE-YOLOv8) model, each incorporating a different FCE module (i.e., modules of Squeeze-and-Excitation (SE), Global Context (GC), Gather-Excite (GE), and Gaussian Context Transformer (GCT)) to enhance the model performance. Experimental results on GRAZPEDWRI-DX dataset demonstrate that our proposed YOLOv8+GC-M3 model improves the mAP@50 value from 65.78% to 66.32%, outperforming the state-of-the-art (SOTA) model while reducing inference time. Furthermore, our proposed YOLOv8+SE-M3 model achieves the highest mAP@50 value of 67.07%, exceeding the SOTA performance. The implementation of this work is available at this https URL.

Title: Don't Stop Me Now: Embedding Based Scheduling for LLMs

• Authors: Rana Shahout, Eran Malach, Chunwei Liu, Weifan Jiang, Minlan Yu, Michael Mitzenmacher
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01035
• Pdf URL: https://arxiv.org/pdf/2410.01035
• Copy Paste: [[2410.01035]] Don't Stop Me Now: Embedding Based Scheduling for LLMs(https://arxiv.org/abs/2410.01035)
• Keywords: large language model
• Abstract: Efficient scheduling is crucial for interactive Large Language Model (LLM) applications, where low request completion time directly impacts user engagement. Size-based scheduling algorithms like Shortest Remaining Process Time (SRPT) aim to reduce average request completion time by leveraging known or estimated request sizes and allowing preemption by incoming jobs with shorter service times. However, two main challenges arise when applying size-based scheduling to LLM systems. First, accurately predicting output lengths from prompts is challenging and often resource-intensive, making it impractical for many systems. As a result, the state-of-the-art LLM systems default to first-come, first-served scheduling, which can lead to head-of-line blocking and reduced system efficiency. Second, preemption introduces extra memory overhead to LLM systems as they must maintain intermediate states for unfinished (preempted) requests. In this paper, we propose TRAIL, a method to obtain output predictions from the target LLM itself. After generating each output token, we recycle the embedding of its internal structure as input for a lightweight classifier that predicts the remaining length for each running request. Using these predictions, we propose a prediction-based SRPT variant with limited preemption designed to account for memory overhead in LLM systems. This variant allows preemption early in request execution when memory consumption is low but restricts preemption as requests approach completion to optimize resource utilization. On the theoretical side, we derive a closed-form formula for this SRPT variant in an M/G/1 queue model, which demonstrates its potential value. In our system, we implement this preemption policy alongside our embedding-based prediction method.

Title: From Facts to Insights: A Study on the Generation and Evaluation of Analytical Reports for Deciphering Earnings Calls

• Authors: Tomas Goldsack, Yang Wang, Chenghua Lin, Chung-Chi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01039
• Pdf URL: https://arxiv.org/pdf/2410.01039
• Copy Paste: [[2410.01039]] From Facts to Insights: A Study on the Generation and Evaluation of Analytical Reports for Deciphering Earnings Calls(https://arxiv.org/abs/2410.01039)
• Keywords: large language model
• Abstract: This paper explores the use of Large Language Models (LLMs) in the generation and evaluation of analytical reports derived from Earnings Calls (ECs). Addressing a current gap in research, we explore the generation of analytical reports with LLMs in a multi-agent framework, designing specialized agents that introduce diverse viewpoints and desirable topics of analysis into the report generation process. Through multiple analyses, we examine the alignment between generated and human-written reports and the impact of both individual and collective agents. Our findings suggest that the introduction of additional agents results in more insightful reports, although reports generated by human experts remain preferred in the majority of cases. Finally, we address the challenging issue of report evaluation, we examine the limitations and strengths of LLMs in assessing the quality of generated reports in different settings, revealing a significant correlation with human experts across multiple dimensions.

Title: Pose Estimation of Buried Deep-Sea Objects using 3D Vision Deep Learning Models

• Authors: Jerry Yan, Chinmay Talegaonkar, Nicholas Antipa, Eric Terrill, Sophia Merrifield
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01061
• Pdf URL: https://arxiv.org/pdf/2410.01061
• Copy Paste: [[2410.01061]] Pose Estimation of Buried Deep-Sea Objects using 3D Vision Deep Learning Models(https://arxiv.org/abs/2410.01061)
• Keywords: transformer, segmentation
• Abstract: We present an approach for pose and burial fraction estimation of debris field barrels found on the seabed in the Southern California San Pedro Basin. Our computational workflow leverages recent advances in foundation models for segmentation and a vision transformer-based approach to estimate the point cloud which defines the geometry of the barrel. We propose BarrelNet for estimating the 6-DOF pose and radius of buried barrels from the barrel point clouds as input. We train BarrelNet using synthetically generated barrel point clouds, and qualitatively demonstrate the potential of our approach using remotely operated vehicle (ROV) video footage of barrels found at a historic dump site. We compare our method to a traditional least squares fitting approach and show significant improvement according to our defined benchmarks.

Title: From Natural Language to SQL: Review of LLM-based Text-to-SQL Systems

• Authors: Ali Mohammadjafari, Anthony S. Maida, Raju Gottumukkala
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01066
• Pdf URL: https://arxiv.org/pdf/2410.01066
• Copy Paste: [[2410.01066]] From Natural Language to SQL: Review of LLM-based Text-to-SQL Systems(https://arxiv.org/abs/2410.01066)
• Keywords: privacy, robust
• Abstract: Since the onset of LLMs, translating natural language queries to structured SQL commands is assuming increasing. Unlike the previous reviews, this survey provides a comprehensive study of the evolution of LLM-based text-to-SQL systems, from early rule-based models to advanced LLM approaches, and how LLMs impacted this field. We discuss benchmarks, evaluation methods and evaluation metrics. Also, we uniquely study the role of integration of knowledge graphs for better contextual accuracy and schema linking in these systems. The current techniques fall into two categories: in-context learning of corpus and fine-tuning, which then leads to approaches such as zero-shot, few-shot learning from the end, and data augmentation. Finally, we highlight key challenges such as computational efficiency, model robustness, and data privacy with perspectives toward their development and improvements in potential areas for future of LLM-based text-to-SQL system.

Title: Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness

• Authors: Eli Chien, Pan Li
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.01068
• Pdf URL: https://arxiv.org/pdf/2410.01068
• Copy Paste: [[2410.01068]] Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness(https://arxiv.org/abs/2410.01068)
• Keywords: privacy
• Abstract: We study the Differential Privacy (DP) guarantee of hidden-state Noisy-SGD algorithms over a bounded domain. Standard privacy analysis for Noisy-SGD assumes all internal states are revealed, which leads to a divergent R'enyi DP bound with respect to the number of iterations. Ye & Shokri (2022) and Altschuler & Talwar (2022) proved convergent bounds for smooth (strongly) convex losses, and raise open questions about whether these assumptions can be relaxed. We provide positive answers by proving convergent R'enyi DP bound for non-convex non-smooth losses, where we show that requiring losses to have Hölder continuous gradient is sufficient. We also provide a strictly better privacy bound compared to state-of-the-art results for smooth strongly convex losses. Our analysis relies on the improvement of shifted divergence analysis in multiple aspects, including forward Wasserstein distance tracking, identifying the optimal shifts allocation, and the H"older reduction lemma. Our results further elucidate the benefit of hidden-state analysis for DP and its applicability.

Title: Inferring Kernel $\epsilon$-Machines: Discovering Structure in Complex Systems

• Authors: Alexandra M. Jurgens, Nicolas Brodu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01076
• Pdf URL: https://arxiv.org/pdf/2410.01076
• Copy Paste: [[2410.01076]] Inferring Kernel $\epsilon$-Machines: Discovering Structure in Complex Systems(https://arxiv.org/abs/2410.01076)
• Keywords: robust, diffusion
• Abstract: Previously, we showed that computational mechanic's causal states -- predictively-equivalent trajectory classes for a stochastic dynamical system -- can be cast into a reproducing kernel Hilbert space. The result is a widely-applicable method that infers causal structure directly from very different kinds of observations and systems. Here, we expand this method to explicitly introduce the causal diffusion components it produces. These encode the kernel causal-state estimates as a set of coordinates in a reduced dimension space. We show how each component extracts predictive features from data and demonstrate their application on four examples: first, a simple pendulum -- an exactly solvable system; second, a molecular-dynamic trajectory of $n$-butane -- a high-dimensional system with a well-studied energy landscape; third, the monthly sunspot sequence -- the longest-running available time series of direct observations; and fourth, multi-year observations of an active crop field -- a set of heterogeneous observations of the same ecosystem taken for over a decade. In this way, we demonstrate that the empirical kernel causal-states algorithm robustly discovers predictive structures for systems with widely varying dimensionality and stochasticity.

Title: Concept Space Alignment in Multilingual LLMs

• Authors: Qiwei Peng, Anders Søgaard
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01079
• Pdf URL: https://arxiv.org/pdf/2410.01079
• Copy Paste: [[2410.01079]] Concept Space Alignment in Multilingual LLMs(https://arxiv.org/abs/2410.01079)
• Keywords: large language model
• Abstract: Multilingual large language models (LLMs) seem to generalize somewhat across languages. We hypothesize this is a result of implicit vector space alignment. Evaluating such alignment, we see that larger models exhibit very high-quality linear alignments between corresponding concepts in different languages. Our experiments show that multilingual LLMs suffer from two familiar weaknesses: generalization works best for languages with similar typology, and for abstract concepts. For some models, e.g., the Llama-2 family of models, prompt-based embeddings align better than word embeddings, but the projections are less linear -- an observation that holds across almost all model families, indicating that some of the implicitly learned alignments are broken somewhat by prompt-based methods.

Title: Deep Nets with Subsampling Layers Unwittingly Discard Useful Activations at Test-Time

• Authors: Chiao-An Yang, Ziwei Liu, Raymond A. Yeh
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01083
• Pdf URL: https://arxiv.org/pdf/2410.01083
• Copy Paste: [[2410.01083]] Deep Nets with Subsampling Layers Unwittingly Discard Useful Activations at Test-Time(https://arxiv.org/abs/2410.01083)
• Keywords: segmentation
• Abstract: Subsampling layers play a crucial role in deep nets by discarding a portion of an activation map to reduce its spatial dimensions. This encourages the deep net to learn higher-level representations. Contrary to this motivation, we hypothesize that the discarded activations are useful and can be incorporated on the fly to improve models' prediction. To validate our hypothesis, we propose a search and aggregate method to find useful activation maps to be used at test time. We applied our approach to the task of image classification and semantic segmentation. Extensive experiments over nine different architectures on multiple datasets show that our method consistently improves model test-time performance, complementing existing test-time augmentation techniques. Our code is available at this https URL.

Title: FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks

• Authors: Peiran Wu, Che Liu, Canyu Chen, Jun Li, Cosmin I. Bercea, Rossella Arcucci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01089
• Pdf URL: https://arxiv.org/pdf/2410.01089
• Copy Paste: [[2410.01089]] FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks(https://arxiv.org/abs/2410.01089)
• Keywords: fair, large language model
• Abstract: Advancements in Multimodal Large Language Models (MLLMs) have significantly improved medical task performance, such as Visual Question Answering (VQA) and Report Generation (RG). However, the fairness of these models across diverse demographic groups remains underexplored, despite its importance in healthcare. This oversight is partly due to the lack of demographic diversity in existing medical multimodal datasets, which complicates the evaluation of fairness. In response, we propose FMBench, the first benchmark designed to evaluate the fairness of MLLMs performance across diverse demographic attributes. FMBench has the following key features: 1: It includes four demographic attributes: race, ethnicity, language, and gender, across two tasks, VQA and RG, under zero-shot settings. 2: Our VQA task is free-form, enhancing real-world applicability and mitigating the biases associated with predefined choices. 3: We utilize both lexical metrics and LLM-based metrics, aligned with clinical evaluations, to assess models not only for linguistic accuracy but also from a clinical perspective. Furthermore, we introduce a new metric, Fairness-Aware Performance (FAP), to evaluate how fairly MLLMs perform across various demographic attributes. We thoroughly evaluate the performance and fairness of eight state-of-the-art open-source MLLMs, including both general and medical MLLMs, ranging from 7B to 26B parameters on the proposed benchmark. We aim for FMBench to assist the research community in refining model evaluation and driving future advancements in the field. All data and code will be released upon acceptance.

Title: Efficient and Private Marginal Reconstruction with Local Non-Negativity

• Authors: Brett Mullins, Miguel Fuentes, Yingtai Xiao, Daniel Kifer, Cameron Musco, Daniel Sheldon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01091
• Pdf URL: https://arxiv.org/pdf/2410.01091
• Copy Paste: [[2410.01091]] Efficient and Private Marginal Reconstruction with Local Non-Negativity(https://arxiv.org/abs/2410.01091)
• Keywords: privacy
• Abstract: Differential privacy is the dominant standard for formal and quantifiable privacy and has been used in major deployments that impact millions of people. Many differentially private algorithms for query release and synthetic data contain steps that reconstruct answers to queries from answers to other queries measured by the mechanism. Reconstruction is an important subproblem for such mechanisms to economize the privacy budget, minimize error on reconstructed answers, and allow for scalability to high-dimensional datasets. In this paper, we introduce a principled and efficient postprocessing method ReM (Residuals-to-Marginals) for reconstructing answers to marginal queries. Our method builds on recent work on efficient mechanisms for marginal query release, based on making measurements using a residual query basis that admits efficient pseudoinversion, which is an important primitive used in reconstruction. An extension GReM-LNN (Gaussian Residuals-to-Marginals with Local Non-negativity) reconstructs marginals under Gaussian noise satisfying consistency and non-negativity, which often reduces error on reconstructed answers. We demonstrate the utility of ReM and GReM-LNN by applying them to improve existing private query answering mechanisms: ResidualPlanner and MWEM.

Title: Semantic Segmentation of Unmanned Aerial Vehicle Remote Sensing Images using SegFormer

• Authors: Vlatko Spasev, Ivica Dimitrovski, Ivan Chorbev, Ivan Kitanovski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01092
• Pdf URL: https://arxiv.org/pdf/2410.01092
• Copy Paste: [[2410.01092]] Semantic Segmentation of Unmanned Aerial Vehicle Remote Sensing Images using SegFormer(https://arxiv.org/abs/2410.01092)
• Keywords: segmentation
• Abstract: The escalating use of Unmanned Aerial Vehicles (UAVs) as remote sensing platforms has garnered considerable attention, proving invaluable for ground object recognition. While satellite remote sensing images face limitations in resolution and weather susceptibility, UAV remote sensing, employing low-speed unmanned aircraft, offers enhanced object resolution and agility. The advent of advanced machine learning techniques has propelled significant strides in image analysis, particularly in semantic segmentation for UAV remote sensing images. This paper evaluates the effectiveness and efficiency of SegFormer, a semantic segmentation framework, for the semantic segmentation of UAV images. SegFormer variants, ranging from real-time (B0) to high-performance (B5) models, are assessed using the UAVid dataset tailored for semantic segmentation tasks. The research details the architecture and training procedures specific to SegFormer in the context of UAV semantic segmentation. Experimental results showcase the model's performance on benchmark dataset, highlighting its ability to accurately delineate objects and land cover features in diverse UAV scenarios, leading to both high efficiency and performance.

Title: Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank

• Authors: Wenhao Zhan, Scott Fujimoto, Zheqing Zhu, Jason D. Lee, Daniel R. Jiang, Yonathan Efroni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01101
• Pdf URL: https://arxiv.org/pdf/2410.01101
• Copy Paste: [[2410.01101]] Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank(https://arxiv.org/abs/2410.01101)
• Keywords: robust
• Abstract: We study the problem of learning an approximate equilibrium in the offline multi-agent reinforcement learning (MARL) setting. We introduce a structural assumption -- the interaction rank -- and establish that functions with low interaction rank are significantly more robust to distribution shift compared to general ones. Leveraging this observation, we demonstrate that utilizing function classes with low interaction rank, when combined with regularization and no-regret learning, admits decentralized, computationally and statistically efficient learning in offline MARL. Our theoretical results are complemented by experiments that showcase the potential of critic architectures with low interaction rank in offline MARL, contrasting with commonly used single-agent value decomposition architectures.

Title: Approximately Aligned Decoding

• Authors: Daniel Melcer, Sujan Gonugondla, Pramuditha Perera, Haifeng Qian, Wen-Hao Chiang, Yanjun Wang, Nihal Jain, Pranav Garg, Xiaofei Ma, Anoop Deoras
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01103
• Pdf URL: https://arxiv.org/pdf/2410.01103
• Copy Paste: [[2410.01103]] Approximately Aligned Decoding(https://arxiv.org/abs/2410.01103)
• Keywords: large language model
• Abstract: It is common to reject undesired outputs of Large Language Models (LLMs); however, current methods to do so require an excessive amount of computation, or severely distort the distribution of outputs. We present a method to balance the distortion of the output distribution with computational efficiency, allowing for the generation of long sequences of text with difficult-to-satisfy constraints, with less amplification of low probability outputs compared to existing methods. We show through a series of experiments that the task-specific performance of our method is comparable to methods that do not distort the output distribution, while being much more computationally efficient.

Title: softmax is not enough (for sharp out-of-distribution)

• Authors: Petar Veličković, Christos Perivolaropoulos, Federico Barbero, Razvan Pascanu
• Subjects: cs.LG, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2410.01104
• Pdf URL: https://arxiv.org/pdf/2410.01104
• Copy Paste: [[2410.01104]] softmax is not enough (for sharp out-of-distribution)(https://arxiv.org/abs/2410.01104)
• Keywords: robust
• Abstract: A key property of reasoning systems is the ability to make sharp decisions on their input data. For contemporary AI systems, a key carrier of sharp behaviour is the softmax function, with its capability to perform differentiable query-key lookups. It is a common belief that the predictive power of networks leveraging softmax arises from "circuits" which sharply perform certain kinds of computations consistently across many diverse inputs. However, for these circuits to be robust, they would need to generalise well to arbitrary valid inputs. In this paper, we dispel this myth: even for tasks as simple as finding the maximum key, any learned circuitry must disperse as the number of items grows at test time. We attribute this to a fundamental limitation of the softmax function to robustly approximate sharp functions, prove this phenomenon theoretically, and propose adaptive temperature as an ad-hoc technique for improving the sharpness of softmax at inference time.

Title: Embedding-based statistical inference on generative models

• Authors: Hayden Helm, Aranyak Acharyya, Brandon Duderstadt, Youngser Park, Carey E. Priebe
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01106
• Pdf URL: https://arxiv.org/pdf/2410.01106
• Copy Paste: [[2410.01106]] Embedding-based statistical inference on generative models(https://arxiv.org/abs/2410.01106)
• Keywords: generative
• Abstract: The recent cohort of publicly available generative models can produce human expert level content across a variety of topics and domains. Given a model in this cohort as a base model, methods such as parameter efficient fine-tuning, in-context learning, and constrained decoding have further increased generative capabilities and improved both computational and data efficiency. Entire collections of derivative models have emerged as a byproduct of these methods and each of these models has a set of associated covariates such as a score on a benchmark, an indicator for if the model has (or had) access to sensitive information, etc. that may or may not be available to the user. For some model-level covariates, it is possible to use "similar" models to predict an unknown covariate. In this paper we extend recent results related to embedding-based representations of generative models -- the data kernel perspective space -- to classical statistical inference settings. We demonstrate that using the perspective space as the basis of a notion of "similar" is effective for multiple model-level inference tasks.

Title: Count of Monte Crypto: Accounting-based Defenses for Cross-Chain Bridges

• Authors: Enze Liu, Elisa Luo, Jian Chen Yan, Katherine Izhikevich, Stewart Grant, Deian Stefan, Geoffrey M Voelker, Stefan Savage
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.01107
• Pdf URL: https://arxiv.org/pdf/2410.01107
• Copy Paste: [[2410.01107]] Count of Monte Crypto: Accounting-based Defenses for Cross-Chain Bridges(https://arxiv.org/abs/2410.01107)
• Keywords: protect, defense, attack
• Abstract: Between 2021 and 2023, crypto assets valued at over \$US2.6 billion were stolen via attacks on "bridges" -- decentralized services designed to allow inter-blockchain exchange. While the individual exploits in each attack vary, a single design flaw underlies them all: the lack of end-to-end value accounting in cross-chain transactions. In this paper, we empirically analyze twenty million transactions used by key bridges during this period. We show that a simple invariant that balances cross-chain inflows and outflows is compatible with legitimate use, yet precisely identifies every known attack (and several likely attacks) in this data. Further, we show that this approach is not only sufficient for post-hoc audits, but can be implemented in-line in existing bridge designs to provide generic protection against a broad array of bridge vulnerabilities.

Title: RobustEMD: Domain Robust Matching for Cross-domain Few-shot Medical Image Segmentation

• Authors: Yazhou Zhu, Minxian Li, Qiaolin Ye, Shidong Wang, Tong Xin, Haofeng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01110
• Pdf URL: https://arxiv.org/pdf/2410.01110
• Copy Paste: [[2410.01110]] RobustEMD: Domain Robust Matching for Cross-domain Few-shot Medical Image Segmentation(https://arxiv.org/abs/2410.01110)
• Keywords: robust, segmentation
• Abstract: Few-shot medical image segmentation (FSMIS) aims to perform the limited annotated data learning in the medical image analysis scope. Despite the progress has been achieved, current FSMIS models are all trained and deployed on the same data domain, as is not consistent with the clinical reality that medical imaging data is always across different data domains (e.g. imaging modalities, institutions and equipment sequences). How to enhance the FSMIS models to generalize well across the different specific medical imaging domains? In this paper, we focus on the matching mechanism of the few-shot semantic segmentation models and introduce an Earth Mover's Distance (EMD) calculation based domain robust matching mechanism for the cross-domain scenario. Specifically, we formulate the EMD transportation process between the foreground support-query features, the texture structure aware weights generation method, which proposes to perform the sobel based image gradient calculation over the nodes, is introduced in the EMD matching flow to restrain the domain relevant nodes. Besides, the point set level distance measurement metric is introduced to calculated the cost for the transportation from support set nodes to query set nodes. To evaluate the performance of our model, we conduct experiments on three scenarios (i.e., cross-modal, cross-sequence and cross-institution), which includes eight medical datasets and involves three body regions, and the results demonstrate that our model achieves the SoTA performance against the compared models.

Title: Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformers

• Authors: Zeyu Michael Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01128
• Pdf URL: https://arxiv.org/pdf/2410.01128
• Copy Paste: [[2410.01128]] Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformers(https://arxiv.org/abs/2410.01128)
• Keywords: security, attack, transformer
• Abstract: Vision Transformers (ViTs) have become popular in computer vision tasks. Backdoor attacks, which trigger undesirable behaviours in models during inference, threaten ViTs' performance, particularly in security-sensitive tasks. Although backdoor defences have been developed for Convolutional Neural Networks (CNNs), they are less effective for ViTs, and defences tailored to ViTs are scarce. To address this, we present Interleaved Ensemble Unlearning (IEU), a method for finetuning clean ViTs on backdoored datasets. In stage 1, a shallow ViT is finetuned to have high confidence on backdoored data and low confidence on clean data. In stage 2, the shallow ViT acts as a ``gate'' to block potentially poisoned data from the defended ViT. This data is added to an unlearn set and asynchronously unlearned via gradient ascent. We demonstrate IEU's effectiveness on three datasets against 11 state-of-the-art backdoor attacks and show its versatility by applying it to different model architectures.

Title: nGPT: Normalized Transformer with Representation Learning on the Hypersphere

• Authors: Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, Boris Ginsburg
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01131
• Pdf URL: https://arxiv.org/pdf/2410.01131
• Copy Paste: [[2410.01131]] nGPT: Normalized Transformer with Representation Learning on the Hypersphere(https://arxiv.org/abs/2410.01131)
• Keywords: transformer
• Abstract: We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.

Title: Explain Like I'm Five: Using LLMs to Improve PDE Surrogate Models with Text

• Authors: Cooper Lorsung, Amir Barati Farimani
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2410.01137
• Pdf URL: https://arxiv.org/pdf/2410.01137
• Copy Paste: [[2410.01137]] Explain Like I'm Five: Using LLMs to Improve PDE Surrogate Models with Text(https://arxiv.org/abs/2410.01137)
• Keywords: large language model
• Abstract: Solving Partial Differential Equations (PDEs) is ubiquitous in science and engineering. Computational complexity and difficulty in writing numerical solvers has motivated the development of machine learning techniques to generate solutions quickly. Many existing methods are purely data driven, relying solely on numerical solution fields, rather than known system information such as boundary conditions and governing equations. However, the recent rise in popularity of Large Language Models (LLMs) has enabled easy integration of text in multimodal machine learning models. In this work, we use pretrained LLMs to integrate various amounts known system information into PDE learning. Our multimodal approach significantly outperforms our baseline model, FactFormer, in both next-step prediction and autoregressive rollout performance on the 2D Heat, Burgers, Navier-Stokes, and Shallow Water equations. Further analysis shows that pretrained LLMs provide highly structured latent space that is consistent with the amount of system information provided through text.

Title: ProxiMix: Enhancing Fairness with Proximity Samples in Subgroups

• Authors: Jingyu Hu, Jun Hong, Mengnan Du, Weiru Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01145
• Pdf URL: https://arxiv.org/pdf/2410.01145
• Copy Paste: [[2410.01145]] ProxiMix: Enhancing Fairness with Proximity Samples in Subgroups(https://arxiv.org/abs/2410.01145)
• Keywords: fair
• Abstract: Many bias mitigation methods have been developed for addressing fairness issues in machine learning. We found that using linear mixup alone, a data augmentation technique, for bias mitigation, can still retain biases present in dataset labels. Research presented in this paper aims to address this issue by proposing a novel pre-processing strategy in which both an existing mixup method and our new bias mitigation algorithm can be utilized to improve the generation of labels of augmented samples, which are proximity aware. Specifically, we proposed ProxiMix which keeps both pairwise and proximity relationships for fairer data augmentation. We conducted thorough experiments with three datasets, three ML models, and different hyperparameters settings. Our experimental results showed the effectiveness of ProxiMix from both fairness of predictions and fairness of recourse perspectives.

Title: Text2PDE: Latent Diffusion Models for Accessible Physics Simulation

• Authors: Anthony Zhou, Zijie Li, Michael Schneier, John R Buchanan Jr, Amir Barati Farimani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01153
• Pdf URL: https://arxiv.org/pdf/2410.01153
• Copy Paste: [[2410.01153]] Text2PDE: Latent Diffusion Models for Accessible Physics Simulation(https://arxiv.org/abs/2410.01153)
• Keywords: diffusion
• Abstract: Recent advances in deep learning have inspired numerous works on data-driven solutions to partial differential equation (PDE) problems. These neural PDE solvers can often be much faster than their numerical counterparts; however, each presents its unique limitations and generally balances training cost, numerical accuracy, and ease of applicability to different problem setups. To address these limitations, we introduce several methods to apply latent diffusion models to physics simulation. Firstly, we introduce a mesh autoencoder to compress arbitrarily discretized PDE data, allowing for efficient diffusion training across various physics. Furthermore, we investigate full spatio-temporal solution generation to mitigate autoregressive error accumulation. Lastly, we investigate conditioning on initial physical quantities, as well as conditioning solely on a text prompt to introduce text2PDE generation. We show that language can be a compact, interpretable, and accurate modality for generating physics simulations, paving the way for more usable and accessible PDE solvers. Through experiments on both uniform and structured grids, we show that the proposed approach is competitive with current neural PDE solvers in both accuracy and efficiency, with promising scaling behavior up to $\sim$3 billion parameters. By introducing a scalable, accurate, and usable physics simulator, we hope to bring neural PDE solvers closer to practical use.

Title: Document Type Classification using File Names

• Authors: Zhijian Li, Stefan Larson, Kevin Leach
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01166
• Pdf URL: https://arxiv.org/pdf/2410.01166
• Copy Paste: [[2410.01166]] Document Type Classification using File Names(https://arxiv.org/abs/2410.01166)
• Keywords: extraction
• Abstract: Rapid document classification is critical in several time-sensitive applications like digital forensics and large-scale media classification. Traditional approaches that rely on heavy-duty deep learning models fall short due to high inference times over vast input datasets and computational resources associated with analyzing whole documents. In this paper, we present a method using lightweight supervised learning models, combined with a TF-IDF feature extraction-based tokenization method, to accurately and efficiently classify documents based solely on file names that substantially reduces inference time. This approach can distinguish ambiguous file names from the indicative file names through confidence scores and through using a negative class representing ambiguous file names. Our results indicate that file name classifiers can process more than 80% of the in-scope data with 96.7% accuracy when tested on a dataset with a large portion of out-of-scope data with respect to the training dataset while being 442.43x faster than more complex models such as DiT. Our method offers a crucial solution for efficiently processing vast datasets in critical scenarios, enabling fast, more reliable document classification.

Title: BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented Generation

• Authors: Bryan Li, Samar Haider, Fiona Luo, Adwait Agashe, Chris Callison-Burch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01171
• Pdf URL: https://arxiv.org/pdf/2410.01171
• Copy Paste: [[2410.01171]] BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented Generation(https://arxiv.org/abs/2410.01171)
• Keywords: robust, large language model
• Abstract: Large language models excel at creative generation but continue to struggle with the issues of hallucination and bias. While retrieval-augmented generation (RAG) provides a framework for grounding LLMs' responses in accurate and up-to-date information, it still raises the question of bias: which sources should be selected for inclusion in the context? And how should their importance be weighted? In this paper, we study the challenge of cross-lingual RAG and present a dataset to investigate the robustness of existing systems at answering queries about geopolitical disputes, which exist at the intersection of linguistic, cultural, and political boundaries. Our dataset is sourced from Wikipedia pages containing information relevant to the given queries and we investigate the impact of including additional context, as well as the composition of this context in terms of language and source, on an LLM's response. Our results show that existing RAG systems continue to be challenged by cross-lingual use cases and suffer from a lack of consistency when they are provided with competing information in multiple languages. We present case studies to illustrate these issues and outline steps for future research to address these challenges. We make our dataset and code publicly available at this https URL.

Title: Towards Inference-time Category-wise Safety Steering for Large Language Models

• Authors: Amrita Bhattacharjee, Shaona Ghosh, Traian Rebedea, Christopher Parisien
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01174
• Pdf URL: https://arxiv.org/pdf/2410.01174
• Copy Paste: [[2410.01174]] Towards Inference-time Category-wise Safety Steering for Large Language Models(https://arxiv.org/abs/2410.01174)
• Keywords: interpretability, large language model
• Abstract: While large language models (LLMs) have seen unprecedented advancements in capabilities and applications across a variety of use-cases, safety alignment of these models is still an area of active research. The fragile nature of LLMs, even models that have undergone extensive alignment and safety training regimes, warrants additional safety steering steps via training-free, inference-time methods. While recent work in the area of mechanistic interpretability has investigated how activations in latent representation spaces may encode concepts, and thereafter performed representation engineering to induce such concepts in LLM outputs, the applicability of such for safety is relatively under-explored. Unlike recent inference-time safety steering works, in this paper we explore safety steering of LLM outputs using: (i) category-specific steering vectors, thereby enabling fine-grained control over the steering, and (ii) sophisticated methods for extracting informative steering vectors for more effective safety steering while retaining quality of the generated text. We demonstrate our exploration on multiple LLMs and datasets, and showcase the effectiveness of the proposed steering method, along with a discussion on the implications and best practices.

Title: UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark

• Authors: Hasnat Md Abdullah, Tian Liu, Kangda Wei, Shu Kong, Ruihong Huang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01180
• Pdf URL: https://arxiv.org/pdf/2410.01180
• Copy Paste: [[2410.01180]] UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark(https://arxiv.org/abs/2410.01180)
• Keywords: large language model
• Abstract: Localizing unusual activities, such as human errors or surveillance incidents, in videos holds practical significance. However, current video understanding models struggle with localizing these unusual events likely because of their insufficient representation in models' pretraining datasets. To explore foundation models' capability in localizing unusual activity, we introduce UAL-Bench, a comprehensive benchmark for unusual activity localization, featuring three video datasets: UAG-OOPS, UAG-SSBD, UAG-FunQA, and an instruction-tune dataset: OOPS-UAG-Instruct, to improve model capabilities. UAL-Bench evaluates three approaches: Video-Language Models (Vid-LLMs), instruction-tuned Vid-LLMs, and a novel integration of Vision-Language Models and Large Language Models (VLM-LLM). Our results show the VLM-LLM approach excels in localizing short-span unusual events and predicting their onset (start time) more accurately than Vid-LLMs. We also propose a new metric, R@1, TD <= p, to address limitations in existing evaluation methods. Our findings highlight the challenges posed by long-duration videos, particularly in autism diagnosis scenarios, and the need for further advancements in localization techniques. Our work not only provides a benchmark for unusual activity localization but also outlines the key challenges for existing foundation models, suggesting future research directions on this important task.

Title: Efficient PAC Learning of Halfspaces with Constant Malicious Noise Rate

• Authors: Xiaoyu Li, Jie Shen
• Subjects: cs.LG, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01186
• Pdf URL: https://arxiv.org/pdf/2410.01186
• Copy Paste: [[2410.01186]] Efficient PAC Learning of Halfspaces with Constant Malicious Noise Rate(https://arxiv.org/abs/2410.01186)
• Keywords: robust
• Abstract: Understanding noise tolerance of learning algorithms under certain conditions is a central quest in learning theory. In this work, we study the problem of computationally efficient PAC learning of halfspaces in the presence of malicious noise, where an adversary can corrupt both instances and labels of training samples. The best-known noise tolerance either depends on a target error rate under distributional assumptions or on a margin parameter under large-margin conditions. In this work, we show that when both types of conditions are satisfied, it is possible to achieve {\em constant} noise tolerance by minimizing a reweighted hinge loss. Our key ingredients include: 1) an efficient algorithm that finds weights to control the gradient deterioration from corrupted samples, and 2) a new analysis on the robustness of the hinge loss equipped with such weights.

Title: Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs

• Authors: Chengyuan Liu, Shihang Wang, Lizhi Qing, Kun Kuang, Yangyang Kang, Changlong Sun, Fei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01188
• Pdf URL: https://arxiv.org/pdf/2410.01188
• Copy Paste: [[2410.01188]] Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs(https://arxiv.org/abs/2410.01188)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) demonstrate impressive generation abilities, they frequently struggle when it comes to specialized domains due to their limited domain-specific knowledge. Studies on domain-specific LLMs resort to expanding the vocabulary before fine-tuning on domain-specific corpus, aiming to decrease the sequence length and enhance efficiency during decoding, without thoroughly investigating the results of vocabulary expansion to LLMs over different domains. Our pilot study reveals that expansion with only a subset of the entire vocabulary may lead to superior performance. Guided by the discovery, this paper explores how to identify a vocabulary subset to achieve the optimal results. We introduce VEGAD, an adaptive method that automatically identifies valuable words from a given domain vocabulary. Our method has been validated through experiments on three Chinese datasets, demonstrating its effectiveness. Additionally, we have undertaken comprehensive analyses of the method. The selection of a optimal subset for expansion has shown to enhance performance on both domain-specific tasks and general tasks, showcasing the potential of VEGAD.

Title: Were RNNs All We Needed?

• Authors: Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, Hossein Hajimirsadegh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01201
• Pdf URL: https://arxiv.org/pdf/2410.01201
• Copy Paste: [[2410.01201]] Were RNNs All We Needed?(https://arxiv.org/abs/2410.01201)
• Keywords: transformer
• Abstract: The scalability limitations of Transformers regarding sequence length have renewed interest in recurrent sequence models that are parallelizable during training. As a result, many novel recurrent architectures, such as S4, Mamba, and Aaren, have been proposed that achieve comparable performance. In this work, we revisit traditional recurrent neural networks (RNNs) from over a decade ago: LSTMs (1997) and GRUs (2014). While these models were slow due to requiring to backpropagate through time (BPTT), we show that by removing their hidden state dependencies from their input, forget, and update gates, LSTMs and GRUs no longer need to BPTT and can be efficiently trained in parallel. Building on this, we introduce minimal versions (minLSTMs and minGRUs) that (1) use significantly fewer parameters than their traditional counterparts and (2) are fully parallelizable during training (175x faster for a sequence of length 512). Lastly, we show that these stripped-down versions of decade-old RNNs match the empirical performance of recent sequence models.

Title: StringLLM: Understanding the String Processing Capability of Large Language Models

• Authors: Xilong Wang, Hao Fu, Neil Zhenqiang Gong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01208
• Pdf URL: https://arxiv.org/pdf/2410.01208
• Copy Paste: [[2410.01208]] StringLLM: Understanding the String Processing Capability of Large Language Models(https://arxiv.org/abs/2410.01208)
• Keywords: large language model
• Abstract: String processing, which mainly involves the analysis and manipulation of strings, is a fundamental component of modern computing. Despite the significant advancements of large language models (LLMs) in various natural language processing (NLP) tasks, their capability in string processing remains underexplored and underdeveloped. To bridge this gap, we present a comprehensive study of LLMs' string processing capability. In particular, we first propose StringLLM, a method to construct datasets for benchmarking string processing capability of LLMs. We use StringLLM to build a series of datasets, referred to as StringBench. It encompasses a wide range of string processing tasks, allowing us to systematically evaluate LLMs' performance in this area. Our evaluations indicate that LLMs struggle with accurately processing strings compared to humans. To uncover the underlying reasons for this limitation, we conduct an in-depth analysis and subsequently propose an effective approach that significantly enhances LLMs' string processing capability via fine-tuning. This work provides a foundation for future research to understand LLMs' string processing capability. Our code and data are available at this https URL.

Title: Debiasing Federated Learning with Correlated Client Participation

• Authors: Zhenyu Sun, Ziyang Zhang, Zheng Xu, Gauri Joshi, Pranay Sharma, Ermin Wei
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.01209
• Pdf URL: https://arxiv.org/pdf/2410.01209
• Copy Paste: [[2410.01209]] Debiasing Federated Learning with Correlated Client Participation(https://arxiv.org/abs/2410.01209)
• Keywords: federate
• Abstract: In cross-device federated learning (FL) with millions of mobile clients, only a small subset of clients participate in training in every communication round, and Federated Averaging (FedAvg) is the most popular algorithm in practice. Existing analyses of FedAvg usually assume the participating clients are independently sampled in each round from a uniform distribution, which does not reflect real-world scenarios. This paper introduces a theoretical framework that models client participation in FL as a Markov chain to study optimization convergence when clients have non-uniform and correlated participation across rounds. We apply this framework to analyze a more general and practical pattern: every client must wait a minimum number of $R$ rounds (minimum separation) before re-participating. We theoretically prove and empirically observe that increasing minimum separation reduces the bias induced by intrinsic non-uniformity of client availability in cross-device FL systems. Furthermore, we develop an effective debiasing algorithm for FedAvg that provably converges to the unbiased optimal solution under arbitrary minimum separation and unknown client availability distribution.

Title: Polyp-SES: Automatic Polyp Segmentation with Self-Enriched Semantic Model

• Authors: Quang Vinh Nguyen, Thanh Hoang Son Vo, Sae-Ryung Kang, Soo-Hyung Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01210
• Pdf URL: https://arxiv.org/pdf/2410.01210
• Copy Paste: [[2410.01210]] Polyp-SES: Automatic Polyp Segmentation with Self-Enriched Semantic Model(https://arxiv.org/abs/2410.01210)
• Keywords: transformer, segmentation
• Abstract: Automatic polyp segmentation is crucial for effective diagnosis and treatment in colonoscopy images. Traditional methods encounter significant challenges in accurately delineating polyps due to limitations in feature representation and the handling of variability in polyp appearance. Deep learning techniques, including CNN and Transformer-based methods, have been explored to improve polyp segmentation accuracy. However, existing approaches often neglect additional semantics, restricting their ability to acquire adequate contexts of polyps in colonoscopy images. In this paper, we propose an innovative method named ``Automatic Polyp Segmentation with Self-Enriched Semantic Model'' to address these limitations. First, we extract a sequence of features from an input image and decode high-level features to generate an initial segmentation mask. Using the proposed self-enriched semantic module, we query potential semantics and augment deep features with additional semantics, thereby aiding the model in understanding context more effectively. Extensive experiments show superior segmentation performance of the proposed method against state-of-the-art polyp segmentation baselines across five polyp benchmarks in both superior learning and generalization capabilities.

Title: From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging

• Authors: Yuling Shi, Songsong Wang, Chengcheng Wan, Xiaodong Gu
• Subjects: cs.CL, cs.AI, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.01215
• Pdf URL: https://arxiv.org/pdf/2410.01215
• Copy Paste: [[2410.01215]] From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging(https://arxiv.org/abs/2410.01215)
• Keywords: robust, large language model
• Abstract: While large language models have made significant strides in code generation, the pass rate of the generated code is bottlenecked on subtle errors, often requiring human intervention to pass tests, especially for complex problems. Existing LLM-based debugging systems treat generated programs as monolithic units, failing to address bugs at multiple levels of granularity, from low-level syntax errors to high-level algorithmic flaws. In this paper, we introduce Multi-Granularity Debugger (MGDebugger), a hierarchical code debugger by isolating, identifying, and resolving bugs at various levels of granularity. MGDebugger decomposes problematic code into a hierarchical tree structure of subfunctions, with each level representing a particular granularity of error. During debugging, it analyzes each subfunction and iteratively resolves bugs in a bottom-up manner. To effectively test each subfunction, we propose an LLM-simulated Python executor, which traces code execution and tracks important variable states to pinpoint errors accurately. Extensive experiments demonstrate that MGDebugger outperforms existing debugging systems, achieving an 18.9% improvement in accuracy over seed generations in HumanEval and a 97.6% repair success rate in HumanEvalFix. Furthermore, MGDebugger effectively fixes bugs across different categories and difficulty levels, demonstrating its robustness and effectiveness.

Title: Perceptual Piercing: Human Visual Cue-based Object Detection in Low Visibility Conditions

• Authors: Ashutosh Kumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01225
• Pdf URL: https://arxiv.org/pdf/2410.01225
• Copy Paste: [[2410.01225]] Perceptual Piercing: Human Visual Cue-based Object Detection in Low Visibility Conditions(https://arxiv.org/abs/2410.01225)
• Keywords: security
• Abstract: This study proposes a novel deep learning framework inspired by atmospheric scattering and human visual cortex mechanisms to enhance object detection under poor visibility scenarios such as fog, smoke, and haze. These conditions pose significant challenges for object recognition, impacting various sectors, including autonomous driving, aviation management, and security systems. The objective is to enhance the precision and reliability of detection systems under adverse environmental conditions. The research investigates the integration of human-like visual cues, particularly focusing on selective attention and environmental adaptability, to ascertain their impact on object detection's computational efficiency and accuracy. This paper proposes a multi-tiered strategy that integrates an initial quick detection process, followed by targeted region-specific dehazing, and concludes with an in-depth detection phase. The approach is validated using the Foggy Cityscapes, RESIDE-beta (OTS and RTTS) datasets and is anticipated to set new performance standards in detection accuracy while significantly optimizing computational efficiency. The findings offer a viable solution for enhancing object detection in poor visibility and contribute to the broader understanding of integrating human visual principles into deep learning algorithms for intricate visual recognition challenges.

Title: Towards Native Generative Model for 3D Head Avatar

• Authors: Yiyu Zhuang, Yuxiao He, Jiawei Zhang, Yanwen Wang, Jiahe Zhu, Yao Yao, Siyu Zhu, Xun Cao, Hao Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01226
• Pdf URL: https://arxiv.org/pdf/2410.01226
• Copy Paste: [[2410.01226]] Towards Native Generative Model for 3D Head Avatar(https://arxiv.org/abs/2410.01226)
• Keywords: generative
• Abstract: Creating 3D head avatars is a significant yet challenging task for many applicated scenarios. Previous studies have set out to learn 3D human head generative models using massive 2D image data. Although these models are highly generalizable for human appearance, their result models are not 360$^\circ$-renderable, and the predicted 3D geometry is unreliable. Therefore, such results cannot be used in VR, game modeling, and other scenarios that require 360$^\circ$-renderable 3D head models. An intuitive idea is that 3D head models with limited amount but high 3D accuracy are more reliable training data for a high-quality 3D generative model. In this vein, we delve into how to learn a native generative model for 360$^\circ$ full head from a limited 3D head dataset. Specifically, three major problems are studied: 1) how to effectively utilize various representations for generating the 360$^\circ$-renderable human head; 2) how to disentangle the appearance, shape, and motion of human faces to generate a 3D head model that can be edited by appearance and driven by motion; 3) and how to extend the generalization capability of the generative model to support downstream tasks. Comprehensive experiments are conducted to verify the effectiveness of the proposed model. We hope the proposed models and artist-designed dataset can inspire future research on learning native generative 3D head models from limited 3D datasets.

Title: Automatic deductive coding in discourse analysis: an application of large language models in learning analytics

• Authors: Lishan Zhang, Han Wu, Xiaoshan Huang, Tengfei Duan, Hanxiang Du
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.01240
• Pdf URL: https://arxiv.org/pdf/2410.01240
• Copy Paste: [[2410.01240]] Automatic deductive coding in discourse analysis: an application of large language models in learning analytics(https://arxiv.org/abs/2410.01240)
• Keywords: large language model
• Abstract: Deductive coding is a common discourse analysis method widely used by learning science and learning analytics researchers for understanding teaching and learning interactions. It often requires researchers to manually label all discourses to be analyzed according to a theoretically guided coding scheme, which is time-consuming and labor-intensive. The emergence of large language models such as GPT has opened a new avenue for automatic deductive coding to overcome the limitations of traditional deductive coding. To evaluate the usefulness of large language models in automatic deductive coding, we employed three different classification methods driven by different artificial intelligence technologies, including the traditional text classification method with text feature engineering, BERT-like pretrained language model and GPT-like pretrained large language model (LLM). We applied these methods to two different datasets and explored the potential of GPT and prompt engineering in automatic deductive coding. By analyzing and comparing the accuracy and Kappa values of these three classification methods, we found that GPT with prompt engineering outperformed the other two methods on both datasets with limited number of training samples. By providing detailed prompt structures, the reported work demonstrated how large language models can be used in the implementation of automatic deductive coding.

Title: AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses

• Authors: Xiaotian Lu, Jiyi Li, Koh Takeuchi, Hisashi Kashima
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01246
• Pdf URL: https://arxiv.org/pdf/2410.01246
• Copy Paste: [[2410.01246]] AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses(https://arxiv.org/abs/2410.01246)
• Keywords: large language model
• Abstract: Question answering (QA) tasks have been extensively studied in the field of natural language processing (NLP). Answers to open-ended questions are highly diverse and difficult to quantify, and cannot be simply evaluated as correct or incorrect, unlike close-ended questions with definitive answers. While large language models (LLMs) have demonstrated strong capabilities across various tasks, they exhibit relatively weaker performance in evaluating answers to open-ended questions. In this study, we propose a method that leverages LLMs and the analytic hierarchy process (AHP) to assess answers to open-ended questions. We utilized LLMs to generate multiple evaluation criteria for a question. Subsequently, answers were subjected to pairwise comparisons under each criterion with LLMs, and scores for each answer were calculated in the AHP. We conducted experiments on four datasets using both ChatGPT-3.5-turbo and GPT-4. Our results indicate that our approach more closely aligns with human judgment compared to the four baselines. Additionally, we explored the impact of the number of criteria, variations in models, and differences in datasets on the results.

Title: HelpSteer2-Preference: Complementing Ratings with Preferences

• Authors: Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, Yi Dong
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01257
• Pdf URL: https://arxiv.org/pdf/2410.01257
• Copy Paste: [[2410.01257]] HelpSteer2-Preference: Complementing Ratings with Preferences(https://arxiv.org/abs/2410.01257)
• Keywords: interpretability
• Abstract: Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style or Regression style. However, there is a lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-written justifications. Using this data, we conduct the first head-to-head comparison of Bradley-Terry and Regression models when adequately matched for data. Based on insights derived from such a comparison, we propose a novel approach to combine Bradley-Terry and Regression reward modeling. A Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. We also demonstrate the effectiveness of this reward model at aligning models to follow instructions in RLHF. We open-source this dataset (CC-BY-4.0 license) at this https URL and openly release the trained Reward Model at this https URL

Title: OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objects

• Authors: Wenmo Qiu, Xinhan Di
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01261
• Pdf URL: https://arxiv.org/pdf/2410.01261
• Copy Paste: [[2410.01261]] OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objects(https://arxiv.org/abs/2410.01261)
• Keywords: large language model
• Abstract: There is a gap in the understanding of occluded objects in existing large-scale visual language multi-modal models. Current state-of-the-art multimodal models fail to provide satisfactory results in describing occluded objects for visual-language multimodal models through universal visual encoders. Another challenge is the limited number of datasets containing image-text pairs with a large number of occluded objects. Therefore, we introduce a novel multimodal model that applies a newly designed visual encoder to understand occluded objects in RGB images. We also introduce a large-scale visual-language pair dataset for training large-scale visual-language multimodal models and understanding occluded objects. We start our experiments comparing with the state-of-the-art models.

Title: Aggregation of Multi Diffusion Models for Enhancing Learned Representations

• Authors: Conghan Yue, Zhengwei Peng, Shiyan Du, Zhi Ji, Dongyu Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01262
• Pdf URL: https://arxiv.org/pdf/2410.01262
• Copy Paste: [[2410.01262]] Aggregation of Multi Diffusion Models for Enhancing Learned Representations(https://arxiv.org/abs/2410.01262)
• Keywords: diffusion
• Abstract: Diffusion models have achieved remarkable success in image generation, particularly with the various applications of classifier-free guidance conditional diffusion models. While many diffusion models perform well when controlling for particular aspect among style, character, and interaction, they struggle with fine-grained control due to dataset limitations and intricate model architecture design. This paper introduces a novel algorithm, Aggregation of Multi Diffusion Models (AMDM), which synthesizes features from multiple diffusion models into a specified model, enhancing its learned representations to activate specific features for fine-grained control. AMDM consists of two key components: spherical aggregation and manifold optimization. Spherical aggregation merges intermediate variables from different diffusion models with minimal manifold deviation, while manifold optimization refines these variables to align with the intermediate data manifold, enhancing sampling quality. Experimental results demonstrate that AMDM significantly improves fine-grained control without additional training or inference time, proving its effectiveness. Additionally, it reveals that diffusion models initially focus on features such as position, attributes, and style, with later stages improving generation quality and consistency. AMDM offers a new perspective for tackling the challenges of fine-grained conditional control generation in diffusion models: We can fully utilize existing conditional diffusion models that control specific aspects, or develop new ones, and then aggregate them using the AMDM algorithm. This eliminates the need for constructing complex datasets, designing intricate model architectures, and incurring high training costs. Code is available at: this https URL

Title: Backdooring Vision-Language Models with Out-Of-Distribution Data

• Authors: Weimin Lyu, Jiachen Yao, Saumya Gupta, Lu Pang, Tao Sun, Lingjie Yi, Lijie Hu, Haibin Ling, Chao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01264
• Pdf URL: https://arxiv.org/pdf/2410.01264
• Copy Paste: [[2410.01264]] Backdooring Vision-Language Models with Out-Of-Distribution Data(https://arxiv.org/abs/2410.01264)
• Keywords: security, attack, large language model
• Abstract: The emergence of Vision-Language Models (VLMs) represents a significant advancement in integrating computer vision with Large Language Models (LLMs) to generate detailed text descriptions from visual inputs. Despite their growing importance, the security of VLMs, particularly against backdoor attacks, is under explored. Moreover, prior works often assume attackers have access to the original training data, which is often unrealistic. In this paper, we address a more practical and challenging scenario where attackers must rely solely on Out-Of-Distribution (OOD) data. We introduce VLOOD (Backdooring Vision-Language Models with Out-of-Distribution Data), a novel approach with two key contributions: (1) demonstrating backdoor attacks on VLMs in complex image-to-text tasks while minimizing degradation of the original semantics under poisoned inputs, and (2) proposing innovative techniques for backdoor injection without requiring any access to the original training data. Our evaluation on image captioning and visual question answering (VQA) tasks confirms the effectiveness of VLOOD, revealing a critical security vulnerability in VLMs and laying the foundation for future research on securing multimodal models against sophisticated threats.

Title: Panopticus: Omnidirectional 3D Object Detection on Resource-constrained Edge Devices

• Authors: Jeho Lee, Chanyoung Jung, Jiwon Kim, Hojung Cha
• Subjects: cs.CV, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.01270
• Pdf URL: https://arxiv.org/pdf/2410.01270
• Copy Paste: [[2410.01270]] Panopticus: Omnidirectional 3D Object Detection on Resource-constrained Edge Devices(https://arxiv.org/abs/2410.01270)
• Keywords: privacy
• Abstract: 3D object detection with omnidirectional views enables safety-critical applications such as mobile robot navigation. Such applications increasingly operate on resource-constrained edge devices, facilitating reliable processing without privacy concerns or network delays. To enable cost-effective deployment, cameras have been widely adopted as a low-cost alternative to LiDAR sensors. However, the compute-intensive workload to achieve high performance of camera-based solutions remains challenging due to the computational limitations of edge devices. In this paper, we present Panopticus, a carefully designed system for omnidirectional and camera-based 3D detection on edge devices. Panopticus employs an adaptive multi-branch detection scheme that accounts for spatial complexities. To optimize the accuracy within latency limits, Panopticus dynamically adjusts the model's architecture and operations based on available edge resources and spatial characteristics. We implemented Panopticus on three edge devices and conducted experiments across real-world environments based on the public self-driving dataset and our mobile 360° camera dataset. Experiment results showed that Panopticus improves accuracy by 62% on average given the strict latency objective of 33ms. Also, Panopticus achieves a 2.1{\times} latency reduction on average compared to baselines.

Title: "No Matter What You Do!": Mitigating Backdoor Attacks in Graph Neural Networks

• Authors: Jiale Zhang, Chengcheng Zhu, Bosen Rao, Hao Sui, Xiaobing Sun, Bing Chen, Chunyi Zhou, Shouling Ji
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01272
• Pdf URL: https://arxiv.org/pdf/2410.01272
• Copy Paste: [[2410.01272]] "No Matter What You Do!": Mitigating Backdoor Attacks in Graph Neural Networks(https://arxiv.org/abs/2410.01272)
• Keywords: defense, attack
• Abstract: Recent studies have exposed that GNNs are vulnerable to several adversarial attacks, among which backdoor attack is one of the toughest. Similar to Deep Neural Networks (DNNs), backdoor attacks in GNNs lie in the fact that the attacker modifies a portion of graph data by embedding triggers and enforces the model to learn the trigger feature during the model training process. Despite the massive prior backdoor defense works on DNNs, defending against backdoor attacks in GNNs is largely unexplored, severely hindering the widespread application of GNNs in real-world tasks. To bridge this gap, we present GCleaner, the first backdoor mitigation method on GNNs. GCleaner can mitigate the presence of the backdoor logic within backdoored GNNs by reversing the backdoor learning procedure, aiming to restore the model performance to a level similar to that is directly trained on the original clean dataset. To achieve this objective, we ask: How to recover universal and hard backdoor triggers in GNNs? How to unlearn the backdoor trigger feature while maintaining the model performance? We conduct the graph trigger recovery via the explanation method to identify optimal trigger locations, facilitating the search of universal and hard backdoor triggers in the feature space of the backdoored model through maximal similarity. Subsequently, we introduce the backdoor unlearning mechanism, which combines knowledge distillation and gradient-based explainable knowledge for fine-grained backdoor erasure. Extensive experimental evaluations on four benchmark datasets demonstrate that GCleaner can reduce the backdoor attack success rate to 10% with only 1% of clean data, and has almost negligible degradation in model performance, which far outperforms the state-of-the-art (SOTA) defense methods.

Title: Deep Unlearn: Benchmarking Machine Unlearning

• Authors: Xavier F. Cadet, Anastasia Borovykh, Mohammad Malekzadeh, Sara Ahmadi-Abhari, Hamed Haddadi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01276
• Pdf URL: https://arxiv.org/pdf/2410.01276
• Copy Paste: [[2410.01276]] Deep Unlearn: Benchmarking Machine Unlearning(https://arxiv.org/abs/2410.01276)
• Keywords: privacy, attack, membership infer, transformer
• Abstract: Machine unlearning (MU) aims to remove the influence of particular data points from the learnable parameters of a trained machine learning model. This is a crucial capability in light of data privacy requirements, trustworthiness, and safety in deployed models. MU is particularly challenging for deep neural networks (DNNs), such as convolutional nets or vision transformers, as such DNNs tend to memorize a notable portion of their training dataset. Nevertheless, the community lacks a rigorous and multifaceted study that looks into the success of MU methods for DNNs. In this paper, we investigate 18 state-of-the-art MU methods across various benchmark datasets and models, with each evaluation conducted over 10 different initializations, a comprehensive evaluation involving MU over 100K models. We show that, with the proper hyperparameters, Masked Small Gradients (MSG) and Convolution Transpose (CT), consistently perform better in terms of model accuracy and run-time efficiency across different models, datasets, and initializations, assessed by population-based membership inference attacks (MIA) and per-sample unlearning likelihood ratio attacks (U-LiRA). Furthermore, our benchmark highlights the fact that comparing a MU method only with commonly used baselines, such as Gradient Ascent (GA) or Successive Random Relabeling (SRL), is inadequate, and we need better baselines like Negative Gradient Plus (NG+) with proper hyperparameter selection.

Title: Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models

• Authors: Can Demircan, Tankred Saanum, Akshay K. Jagadish, Marcel Binz, Eric Schulz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01280
• Pdf URL: https://arxiv.org/pdf/2410.01280
• Copy Paste: [[2410.01280]] Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models(https://arxiv.org/abs/2410.01280)
• Keywords: large language model
• Abstract: In-context learning, the ability to adapt based on a few examples in the input prompt, is a ubiquitous feature of large language models (LLMs). However, as LLMs' in-context learning abilities continue to improve, understanding this phenomenon mechanistically becomes increasingly important. In particular, it is not well-understood how LLMs learn to solve specific classes of problems, such as reinforcement learning (RL) problems, in-context. Through three different tasks, we first show that Llama $3$ $70$B can solve simple RL problems in-context. We then analyze the residual stream of Llama using Sparse Autoencoders (SAEs) and find representations that closely match temporal difference (TD) errors. Notably, these representations emerge despite the model only being trained to predict the next token. We verify that these representations are indeed causally involved in the computation of TD errors and $Q$-values by performing carefully designed interventions on them. Taken together, our work establishes a methodology for studying and manipulating in-context learning with SAEs, paving the way for a more mechanistic understanding.

Title: Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration

• Authors: Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01285
• Pdf URL: https://arxiv.org/pdf/2410.01285
• Copy Paste: [[2410.01285]] Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration(https://arxiv.org/abs/2410.01285)
• Keywords: protect, large language model
• Abstract: The black-box nature of large language models (LLMs) poses challenges in interpreting results, impacting issues such as data intellectual property protection and hallucination tracing. Training data attribution (TDA) methods are considered effective solutions to address these challenges. Most recent TDA methods rely on influence functions, assuming the model achieves minimized empirical risk. However, achieving this criterion is difficult, and sourcing accuracy can be compromised by fitting errors during model training. In this paper, we introduce a novel TDA method called Debias and Denoise Attribution (DDA), which enhances influence functions by addressing fitting errors. Specifically, the debias strategy seeks to improve the performance of influence functions by eliminating the knowledge bias present in the base model before fine-tuning, while the denoise strategy aims to reduce discrepancies in influence scores arising from varying degrees of fitting during the training process through smoothing techniques. Experimental results demonstrate that our method significantly outperforms existing approaches, achieving an averaged AUC of 91.64%. Moreover, DDA exhibits strong generality and scalability across various sources and different-scale models like LLaMA2, QWEN2, and Mistral.

Title: Mitigating Copy Bias in In-Context Learning through Neuron Pruning

• Authors: Ameen Ali, Lior Wolf, Ivan Titov
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01288
• Pdf URL: https://arxiv.org/pdf/2410.01288
• Copy Paste: [[2410.01288]] Mitigating Copy Bias in In-Context Learning through Neuron Pruning(https://arxiv.org/abs/2410.01288)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) have demonstrated impressive few-shot in-context learning (ICL) abilities. Still, we show that they are sometimes prone to a `copying bias', where they copy answers from provided examples instead of learning the underlying patterns. In this work, we propose a novel and simple method to mitigate such copying bias. First, we create a synthetic task and use the Integrated Gradients method to identify neurons that prioritize copying over generalization. We demonstrate that pruning these neurons consistently improves performance across a diverse set of ICL tasks. We also show that our method is applicable across various LLM architectures, including Transformers and State-Space Models, without requiring modifications. In our analysis, we adopt a task-recognition perspective on ICL and examine task vectors (Hendel et al., 2023) induced by the model. We find that pruning enhances the quality of these vectors, suggesting that the pruned neurons previously hindered effective task recognition.

Title: SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images using a Synthetically-trained Network

• Authors: Ahmed Tawfik Aboukhadra, Nadia Robertini, Jameel Malik, Ahmed Elhayek, Gerd Reis, Didier Stricker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01293
• Pdf URL: https://arxiv.org/pdf/2410.01293
• Copy Paste: [[2410.01293]] SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images using a Synthetically-trained Network(https://arxiv.org/abs/2410.01293)
• Keywords: transformer
• Abstract: Surgery monitoring in Mixed Reality (MR) environments has recently received substantial focus due to its importance in image-based decisions, skill assessment, and robot-assisted surgery. Tracking hands and articulated surgical instruments is crucial for the success of these applications. Due to the lack of annotated datasets and the complexity of the task, only a few works have addressed this problem. In this work, we present SurgeoNet, a real-time neural network pipeline to accurately detect and track surgical instruments from a stereo VR view. Our multi-stage approach is inspired by state-of-the-art neural-network architectural design, like YOLO and Transformers. We demonstrate the generalization capabilities of SurgeoNet in challenging real-world scenarios, achieved solely through training on synthetic data. The approach can be easily extended to any new set of articulated surgical instruments. SurgeoNet's code and data are publicly available.

Title: Endless Jailbreaks with Bijection Learning

• Authors: Brian R.Y. Huang, Maximilian Li, Leonard Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01294
• Pdf URL: https://arxiv.org/pdf/2410.01294
• Copy Paste: [[2410.01294]] Endless Jailbreaks with Bijection Learning(https://arxiv.org/abs/2410.01294)
• Keywords: attack
• Abstract: Despite extensive safety training, LLMs are vulnerable to adversarial inputs. In this work, we introduce a simple but powerful attack paradigm, bijection learning, that yields a practically endless set of jailbreak prompts. We exploit language models' advanced reasoning capabilities to teach them invertible languages (bijections) in context, pass encoded queries to the model to bypass built-in safety mechanisms, and finally decode responses back into English, yielding helpful replies to harmful requests. Our approach proves effective on a wide range of frontier language models and harm categories. Bijection learning is an automated and universal attack that grows stronger with scale: larger models with more advanced reasoning capabilities are more susceptible to bijection learning jailbreaks despite stronger safety mechanisms.

Title: LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion

• Authors: Biao Zhang, Peter Wonka
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.01295
• Pdf URL: https://arxiv.org/pdf/2410.01295
• Copy Paste: [[2410.01295]] LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion(https://arxiv.org/abs/2410.01295)
• Keywords: diffusion, generative
• Abstract: This paper introduces a novel hierarchical autoencoder that maps 3D models into a highly compressed latent space. The hierarchical autoencoder is specifically designed to tackle the challenges arising from large-scale datasets and generative modeling using diffusion. Different from previous approaches that only work on a regular image or volume grid, our hierarchical autoencoder operates on unordered sets of vectors. Each level of the autoencoder controls different geometric levels of detail. We show that the model can be used to represent a wide range of 3D models while faithfully representing high-resolution geometry details. The training of the new architecture takes 0.70x time and 0.58x memory compared to the baseline. We also explore how the new representation can be used for generative modeling. Specifically, we propose a cascaded diffusion framework where each stage is conditioned on the previous stage. Our design extends existing cascaded designs for image and volume grids to vector sets.

Title: Speculative Coreset Selection for Task-Specific Fine-tuning

• Authors: Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Chao Shen, Tianlin Li, Weipeng Jiang, Yang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01296
• Pdf URL: https://arxiv.org/pdf/2410.01296
• Copy Paste: [[2410.01296]] Speculative Coreset Selection for Task-Specific Fine-tuning(https://arxiv.org/abs/2410.01296)
• Keywords: large language model
• Abstract: Task-specific fine-tuning is essential for the deployment of large language models (LLMs), but it requires significant computational resources and time. Existing solutions have proposed coreset selection methods to improve data efficiency and reduce model training overhead, but they still have limitations: 1) Overlooking valuable samples at high pruning rates, which degrades the coreset's performance. 2) Requiring high time overhead during coreset selection to fine-tune and evaluate the target LLM. In this paper, we introduce STAFF, a speculative coreset selection method. STAFF leverages a small model from the same family as the target LLM to efficiently estimate data scores and then verifies the scores on the target LLM to accurately identify and allocate more selection budget to important regions while maintaining coverage of easy regions. We evaluate STAFF on three LLMs and three downstream tasks and show that STAFF improves the performance of SOTA methods by up to 54.3% and reduces selection overhead by up to 70.5% at different pruning rates. Furthermore, we observe that the coreset selected by STAFF at low pruning rates (i.e., 20%) can even obtain better fine-tuning performance than the full dataset.

Title: Revisiting Hierarchical Text Classification: Inference and Metrics

• Authors: Roman Plaud, Matthieu Labeau, Antoine Saillenfest, Thomas Bonald
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01305
• Pdf URL: https://arxiv.org/pdf/2410.01305
• Copy Paste: [[2410.01305]] Revisiting Hierarchical Text Classification: Inference and Metrics(https://arxiv.org/abs/2410.01305)
• Keywords: fair
• Abstract: Hierarchical text classification (HTC) is the task of assigning labels to a text within a structured space organized as a hierarchy. Recent works treat HTC as a conventional multilabel classification problem, therefore evaluating it as such. We instead propose to evaluate models based on specifically designed hierarchical metrics and we demonstrate the intricacy of metric choice and prediction inference method. We introduce a new challenging dataset and we evaluate fairly, recent sophisticated models, comparing them with a range of simple but strong baselines, including a new theoretically motivated loss. Finally, we show that those baselines are very often competitive with the latest models. This highlights the importance of carefully considering the evaluation methodology when proposing new methods for HTC. Code implementation and dataset are available at \url{this https URL}.

Title: Emotion-Aware Response Generation Using Affect-Enriched Embeddings with LLMs

• Authors: Abdur Rasool, Muhammad Irfan Shahzad, Hafsa Aslam, Vincent Chan
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.01306
• Pdf URL: https://arxiv.org/pdf/2410.01306
• Copy Paste: [[2410.01306]] Emotion-Aware Response Generation Using Affect-Enriched Embeddings with LLMs(https://arxiv.org/abs/2410.01306)
• Keywords: large language model
• Abstract: There is a need for empathetic and coherent responses in automated chatbot-facilitated psychotherapy sessions. This study addresses the challenge of enhancing the emotional and contextual understanding of large language models (LLMs) in psychiatric applications. We introduce a novel framework that integrates multiple emotion lexicons, including NRC Emotion Lexicon, VADER, WordNet, and SentiWordNet, with state-of-the-art LLMs such as LLAMA 2, Flan-T5, ChatGPT 3.0, and ChatGPT 4.0. The primary dataset comprises over 2,000 therapy session transcripts from the Counseling and Psychotherapy database, covering discussions on anxiety, depression, trauma, and addiction. We segment the transcripts into smaller chunks, enhancing them with lexical features and computing embeddings using BERT, GPT-3, and RoBERTa to capture semantic and emotional nuances. These embeddings are stored in a FAISS vector database, enabling efficient similarity search and clustering based on cosine similarity. Upon user query, the most relevant segments are retrieved and provided as context to the LLMs, significantly improving the models' ability to generate empathetic and contextually appropriate responses. Experimental evaluations demonstrate that in-corporating emotion lexicons enhances empathy, coherence, informativeness, and fluency scores. Our findings highlight the critical role of emotional embeddings in improving LLM performance for psychotherapy.

Title: Sampling from Energy-based Policies using Diffusion

• Authors: Vineet Jain, Tara Akhound-Sadegh, Siamak Ravanbakhsh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01312
• Pdf URL: https://arxiv.org/pdf/2410.01312
• Copy Paste: [[2410.01312]] Sampling from Energy-based Policies using Diffusion(https://arxiv.org/abs/2410.01312)
• Keywords: diffusion
• Abstract: Energy-based policies offer a flexible framework for modeling complex, multimodal behaviors in reinforcement learning (RL). In maximum entropy RL, the optimal policy is a Boltzmann distribution derived from the soft Q-function, but direct sampling from this distribution in continuous action spaces is computationally intractable. As a result, existing methods typically use simpler parametric distributions, like Gaussians, for policy representation - limiting their ability to capture the full complexity of multimodal action distributions. In this paper, we introduce a diffusion-based approach for sampling from energy-based policies, where the negative Q-function defines the energy function. Based on this approach, we propose an actor-critic method called Diffusion Q-Sampling (DQS) that enables more expressive policy representations, allowing stable learning in diverse environments. We show that our approach enhances exploration and captures multimodal behavior in continuous control tasks, addressing key limitations of existing methods.

Title: Forte : Finding Outliers with Representation Typicality Estimation

• Authors: Debargha Ganguly, Warren Morningstar, Andrew Yu, Vipin Chaudhary
• Subjects: cs.LG, cs.AI, cs.CV, cs.IT
• Abstract URL: https://arxiv.org/abs/2410.01322
• Pdf URL: https://arxiv.org/pdf/2410.01322
• Copy Paste: [[2410.01322]] Forte : Finding Outliers with Representation Typicality Estimation(https://arxiv.org/abs/2410.01322)
• Keywords: generative
• Abstract: Generative models can now produce photorealistic synthetic data which is virtually indistinguishable from the real data used to train it. This is a significant evolution over previous models which could produce reasonable facsimiles of the training data, but ones which could be visually distinguished from the training data by human evaluation. Recent work on OOD detection has raised doubts that generative model likelihoods are optimal OOD detectors due to issues involving likelihood misestimation, entropy in the generative process, and typicality. We speculate that generative OOD detectors also failed because their models focused on the pixels rather than the semantic content of the data, leading to failures in near-OOD cases where the pixels may be similar but the information content is significantly different. We hypothesize that estimating typical sets using self-supervised learners leads to better OOD detectors. We introduce a novel approach that leverages representation learning, and informative summary statistics based on manifold estimation, to address all of the aforementioned issues. Our method outperforms other unsupervised approaches and achieves state-of-the art performance on well-established challenging benchmarks, and new synthetic data detection tasks.

Title: Fair Class-Incremental Learning using Sample Weighting

• Authors: Jaeyoung Park, Minsu Kim, Steven Euijong Whang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01324
• Pdf URL: https://arxiv.org/pdf/2410.01324
• Copy Paste: [[2410.01324]] Fair Class-Incremental Learning using Sample Weighting(https://arxiv.org/abs/2410.01324)
• Keywords: fair
• Abstract: Model fairness is becoming important in class-incremental learning for Trustworthy AI. While accuracy has been a central focus in class-incremental learning, fairness has been relatively understudied. However, naively using all the samples of the current task for training results in unfair catastrophic forgetting for certain sensitive groups including classes. We theoretically analyze that forgetting occurs if the average gradient vector of the current task data is in an "opposite direction" compared to the average gradient vector of a sensitive group, which means their inner products are negative. We then propose a fair class-incremental learning framework that adjusts the training weights of current task samples to change the direction of the average gradient vector and thus reduce the forgetting of underperforming groups and achieve fairness. For various group fairness measures, we formulate optimization problems to minimize the overall losses of sensitive groups while minimizing the disparities among them. We also show the problems can be solved with linear programming and propose an efficient Fairness-aware Sample Weighting (FSW) algorithm. Experiments show that FSW achieves better accuracy-fairness tradeoff results than state-of-the-art approaches on real datasets.

Title: Unveiling Language Skills under Circuits

• Authors: Hang Chen, Jiaying Zhu, Xinyu Yang, Wenya Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01334
• Pdf URL: https://arxiv.org/pdf/2410.01334
• Copy Paste: [[2410.01334]] Unveiling Language Skills under Circuits(https://arxiv.org/abs/2410.01334)
• Keywords: interpretability, transformer
• Abstract: The exploration of language skills in language models (LMs) has always been one of the central goals in mechanistic interpretability. However, existing circuit analyses often fall short in representing the full functional scope of these models, primarily due to the exclusion of Feed-Forward layers. Additionally, isolating the effect of a single language skill from a text, which inherently involves multiple entangled skills, poses a significant challenge. To address these gaps, we introduce a novel concept, Memory Circuit, a minimum unit that fully and independently manipulates the memory-reading functionality of a language model, and disentangle the transformer model precisely into a circuit graph which is an ensemble of paths connecting different memory circuits. Based on this disentanglement, we identify salient circuit paths, named as skill paths, responsible for three crucial language skills, i.e., the Previous Token Skill, Induction Skill and In-Context Learning (ICL) Skill, leveraging causal effect estimation through interventions and counterfactuals. Our experiments on various datasets confirm the correspondence between our identified skill paths and language skills, and validate three longstanding hypotheses: 1) Language skills are identifiable through circuit dissection; 2) Simple language skills reside in shallow layers, whereas complex language skills are found in deeper layers; 3) Complex language skills are formed on top of simpler language skills. Our codes are available at: this https URL.

Title: Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models

• Authors: Lucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou, Nayan Singhal, Hongjiang Lv, Bing Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01335
• Pdf URL: https://arxiv.org/pdf/2410.01335
• Copy Paste: [[2410.01335]] Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models(https://arxiv.org/abs/2410.01335)
• Keywords: transformer, large language model
• Abstract: Model merging, such as model souping, is the practice of combining different models with the same architecture together without further training. In this work, we present a model merging methodology that addresses the difficulty of fine-tuning Large Language Models (LLMs) for target tasks in non-English languages, where task-specific data is often unavailable. We focus on mathematical reasoning and without in-language math data, facilitate cross-lingual transfer by composing language and math capabilities. Starting from the same pretrained model, we fine-tune separate "experts" on math instruction data in English and on generic instruction data in the target language. We then replace the top and bottom transformer layers of the math expert directly with layers from the language expert, which consequently enhances math performance in the target language. The resulting merged models outperform the individual experts and other merging methods on the math benchmark, MGSM, by 10% across four major languages where math instruction data is scarce. In addition, this layer swapping is simple, inexpensive, and intuitive, as it is based on an interpretative analysis of the most important parameter changes during the fine-tuning of each expert. The ability to successfully re-compose LLMs for cross-lingual transfer in this manner opens up future possibilities to combine model expertise, create modular solutions, and transfer reasoning capabilities across languages all post hoc.

Title: VectorGraphNET: Graph Attention Networks for Accurate Segmentation of Complex Technical Drawings

• Authors: Andrea Carrara, Stavros Nousias, André Borrmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01336
• Pdf URL: https://arxiv.org/pdf/2410.01336
• Copy Paste: [[2410.01336]] VectorGraphNET: Graph Attention Networks for Accurate Segmentation of Complex Technical Drawings(https://arxiv.org/abs/2410.01336)
• Keywords: transformer, segmentation
• Abstract: This paper introduces a new approach to extract and analyze vector data from technical drawings in PDF format. Our method involves converting PDF files into SVG format and creating a feature-rich graph representation, which captures the relationships between vector entities using geometrical information. We then apply a graph attention transformer with hierarchical label definition to achieve accurate line-level segmentation. Our approach is evaluated on two datasets, including the public FloorplanCAD dataset, which achieves state-of-the-art results on weighted F1 score, surpassing existing methods. The proposed vector-based method offers a more scalable solution for large-scale technical drawing analysis compared to vision-based approaches, while also requiring significantly less GPU power than current state-of-the-art vector-based techniques. Moreover, it demonstrates improved performance in terms of the weighted F1 (wF1) score on the semantic segmentation task. Our results demonstrate the effectiveness of our approach in extracting meaningful information from technical drawings, enabling new applications, and improving existing workflows in the AEC industry. Potential applications of our approach include automated building information modeling (BIM) and construction planning, which could significantly impact the efficiency and productivity of the industry.

Title: PhyMPGN: Physics-encoded Message Passing Graph Network for spatiotemporal PDE systems

• Authors: Bocheng Zeng, Qi Wang, Mengtao Yan, Yang Liu, Ruizhi Chengze, Yi Zhang, Hongsheng Liu, Zidong Wang, Hao Sun
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2410.01337
• Pdf URL: https://arxiv.org/pdf/2410.01337
• Copy Paste: [[2410.01337]] PhyMPGN: Physics-encoded Message Passing Graph Network for spatiotemporal PDE systems(https://arxiv.org/abs/2410.01337)
• Keywords: diffusion
• Abstract: Solving partial differential equations (PDEs) serves as a cornerstone for modeling complex dynamical systems. Recent progresses have demonstrated grand benefits of data-driven neural-based models for predicting spatiotemporal dynamics (e.g., tremendous speedup gain compared with classical numerical methods). However, most existing neural models rely on rich training data, have limited extrapolation and generalization abilities, and suffer to produce precise or reliable physical prediction under intricate conditions (e.g., irregular mesh or geometry, complex boundary conditions, diverse PDE parameters, etc.). To this end, we propose a new graph learning approach, namely, Physics-encoded Message Passing Graph Network (PhyMPGN), to model spatiotemporal PDE systems on irregular meshes given small training datasets. Specifically, we incorporate a GNN into a numerical integrator to approximate the temporal marching of spatiotemporal dynamics for a given PDE system. Considering that many physical phenomena are governed by diffusion processes, we further design a learnable Laplace block, which encodes the discrete Laplace-Beltrami operator, to aid and guide the GNN learning in a physically feasible solution space. A boundary condition padding strategy is also designed to improve the model convergence and accuracy. Extensive experiments demonstrate that PhyMPGN is capable of accurately predicting various types of spatiotemporal dynamics on coarse unstructured meshes, consistently achieves the state-of-the-art results, and outperforms other baselines with considerable gains.

Title: Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation

• Authors: Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Fanman Meng, Qingbo Wu, Hongliang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01341
• Pdf URL: https://arxiv.org/pdf/2410.01341
• Copy Paste: [[2410.01341]] Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation(https://arxiv.org/abs/2410.01341)
• Keywords: segmentation
• Abstract: In this paper, we explore a novel Text-supervised Egocentic Semantic Segmentation (TESS) task that aims to assign pixel-level categories to egocentric images weakly supervised by texts from image-level labels. In this task with prospective potential, the egocentric scenes contain dense wearer-object relations and inter-object interference. However, most recent third-view methods leverage the frozen Contrastive Language-Image Pre-training (CLIP) model, which is pre-trained on the semantic-oriented third-view data and lapses in the egocentric view due to the ``relation insensitive" problem. Hence, we propose a Cognition Transferring and Decoupling Network (CTDN) that first learns the egocentric wearer-object relations via correlating the image and text. Besides, a Cognition Transferring Module (CTM) is developed to distill the cognitive knowledge from the large-scale pre-trained model to our model for recognizing egocentric objects with various semantics. Based on the transferred cognition, the Foreground-background Decoupling Module (FDM) disentangles the visual representations to explicitly discriminate the foreground and background regions to mitigate false activation areas caused by foreground-background interferential objects during egocentric relation learning. Extensive experiments on four TESS benchmarks demonstrate the effectiveness of our approach, which outperforms many recent related methods by a large margin. Code will be available at this https URL.

Title: Assisted Data Annotation for Business Process Information Extraction from Textual Documents

• Authors: Julian Neuberger, Han van der Aa, Lars Ackermann, Daniel Buschek, Jannic Herrmann, Stefan Jablonski
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01356
• Pdf URL: https://arxiv.org/pdf/2410.01356
• Copy Paste: [[2410.01356]] Assisted Data Annotation for Business Process Information Extraction from Textual Documents(https://arxiv.org/abs/2410.01356)
• Keywords: extraction
• Abstract: Machine-learning based generation of process models from natural language text process descriptions provides a solution for the time-intensive and expensive process discovery phase. Many organizations have to carry out this phase, before they can utilize business process management and its benefits. Yet, research towards this is severely restrained by an apparent lack of large and high-quality datasets. This lack of data can be attributed to, among other things, an absence of proper tool assistance for dataset creation, resulting in high workloads and inferior data quality. We explore two assistance features to support dataset creation, a recommendation system for identifying process information in the text and visualization of the current state of already identified process information as a graphical business process model. A controlled user study with 31 participants shows that assisting dataset creators with recommendations lowers all aspects of workload, up to $-51.0\%$, and significantly improves annotation quality, up to $+38.9\%$. We make all data and code available to encourage further research on additional novel assistance strategies.

Title: FlashMask: Efficient and Rich Mask Extension of FlashAttention

• Authors: Guoxia Wang, Jinle Zeng, Xiyuan Xiao, Siming Wu, Jiabin Yang, Lujing Zheng, Zeyu Chen, Jiang Bian, Dianhai Yu, Haifeng Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01359
• Pdf URL: https://arxiv.org/pdf/2410.01359
• Copy Paste: [[2410.01359]] FlashMask: Efficient and Rich Mask Extension of FlashAttention(https://arxiv.org/abs/2410.01359)
• Keywords: transformer
• Abstract: The computational and memory demands of vanilla attention scale quadratically with the sequence length $N$, posing significant challenges for processing long sequences in Transformer models. FlashAttention alleviates these challenges by eliminating the $O(N^2)$ memory dependency and reducing attention latency through IO-aware memory optimizations. However, its native support for certain attention mask types is limited, and it does not inherently accommodate more complex masking requirements. Previous approaches resort to using dense masks with $O(N^2)$ memory complexity, leading to inefficiencies. In this paper, we propose FlashMask, an extension of FlashAttention that introduces a column-wise sparse representation of attention masks. This approach efficiently represents a wide range of mask types and facilitates the development of optimized kernel implementations. By adopting this novel representation, FlashMask achieves linear memory complexity $O(N)$, suitable for modeling long-context sequences. Moreover, this representation enables kernel optimizations that eliminate unnecessary computations by leveraging sparsity in the attention mask, without sacrificing computational accuracy, resulting in higher computational efficiency. We evaluate FlashMask's performance in fine-tuning and alignment training of LLMs such as SFT, LoRA, DPO, and RM. FlashMask achieves significant throughput improvements, with end-to-end speedups ranging from 1.65x to 3.22x compared to existing FlashAttention dense method. Additionally, our kernel-level comparisons demonstrate that FlashMask surpasses the latest counterpart, FlexAttention, by 12.1% to 60.7% in terms of kernel TFLOPs/s, achieving 37.8% to 62.3% of the theoretical maximum FLOPs/s on the A100 GPU. The code is open-sourced on PaddlePaddle and integrated into PaddleNLP, supporting models with over 100 billion parameters for contexts up to 128K tokens.

Title: PCQPR: Proactive Conversational Question Planning with Reflection

• Authors: Shasha Guo, Lizi Liao, Jing Zhang, Cuiping Li, Hong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01363
• Pdf URL: https://arxiv.org/pdf/2410.01363
• Copy Paste: [[2410.01363]] PCQPR: Proactive Conversational Question Planning with Reflection(https://arxiv.org/abs/2410.01363)
• Keywords: large language model
• Abstract: Conversational Question Generation (CQG) enhances the interactivity of conversational question-answering systems in fields such as education, customer service, and entertainment. However, traditional CQG, focusing primarily on the immediate context, lacks the conversational foresight necessary to guide conversations toward specified conclusions. This limitation significantly restricts their ability to achieve conclusion-oriented conversational outcomes. In this work, we redefine the CQG task as Conclusion-driven Conversational Question Generation (CCQG) by focusing on proactivity, not merely reacting to the unfolding conversation but actively steering it towards a conclusion-oriented question-answer pair. To address this, we propose a novel approach, called Proactive Conversational Question Planning with self-Refining (PCQPR). Concretely, by integrating a planning algorithm inspired by Monte Carlo Tree Search (MCTS) with the analytical capabilities of large language models (LLMs), PCQPR predicts future conversation turns and continuously refines its questioning strategies. This iterative self-refining mechanism ensures the generation of contextually relevant questions strategically devised to reach a specified outcome. Our extensive evaluations demonstrate that PCQPR significantly surpasses existing CQG methods, marking a paradigm shift towards conclusion-oriented conversational question-answering systems.

Title: Harnessing the Latent Diffusion Model for Training-Free Image Style Transfer

• Authors: Kento Masui, Mayu Otani, Masahiro Nomura, Hideki Nakayama
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.01366
• Pdf URL: https://arxiv.org/pdf/2410.01366
• Copy Paste: [[2410.01366]] Harnessing the Latent Diffusion Model for Training-Free Image Style Transfer(https://arxiv.org/abs/2410.01366)
• Keywords: diffusion
• Abstract: Diffusion models have recently shown the ability to generate high-quality images. However, controlling its generation process still poses challenges. The image style transfer task is one of those challenges that transfers the visual attributes of a style image to another content image. Typical obstacle of this task is the requirement of additional training of a pre-trained model. We propose a training-free style transfer algorithm, Style Tracking Reverse Diffusion Process (STRDP) for a pretrained Latent Diffusion Model (LDM). Our algorithm employs Adaptive Instance Normalization (AdaIN) function in a distinct manner during the reverse diffusion process of an LDM while tracking the encoding history of the style image. This algorithm enables style transfer in the latent space of LDM for reduced computational cost, and provides compatibility for various LDM models. Through a series of experiments and a user study, we show that our method can quickly transfer the style of an image without additional training. The speed, compatibility, and training-free aspect of our algorithm facilitates agile experiments with combinations of styles and LDMs for extensive application.

Title: Towards Dynamic Graph Neural Networks with Provably High-Order Expressive Power

• Authors: Zhe Wang, Tianjian Zhao, Zhen Zhang, Jiawei Chen, Sheng Zhou, Yan Feng, Chun Chen, Can Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01367
• Pdf URL: https://arxiv.org/pdf/2410.01367
• Copy Paste: [[2410.01367]] Towards Dynamic Graph Neural Networks with Provably High-Order Expressive Power(https://arxiv.org/abs/2410.01367)
• Keywords: transformer
• Abstract: Dynamic Graph Neural Networks (DyGNNs) have garnered increasing research attention for learning representations on evolving graphs. Despite their effectiveness, the limited expressive power of existing DyGNNs hinders them from capturing important evolving patterns of dynamic graphs. Although some works attempt to enhance expressive capability with heuristic features, there remains a lack of DyGNN frameworks with provable and quantifiable high-order expressive power. To address this research gap, we firstly propose the k-dimensional Dynamic WL tests (k-DWL) as the referencing algorithms to quantify the expressive power of DyGNNs. We demonstrate that the expressive power of existing DyGNNs is upper bounded by the 1-DWL test. To enhance the expressive power, we propose Dynamic Graph Neural Network with High-order expressive power (HopeDGN), which updates the representation of central node pair by aggregating the interaction history with neighboring node pairs. Our theoretical results demonstrate that HopeDGN can achieve expressive power equivalent to the 2-DWL test. We then present a Transformer-based implementation for the local variant of HopeDGN. Experimental results show that HopeDGN achieved performance improvements of up to 3.12%, demonstrating the effectiveness of HopeDGN.

Title: Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systems

• Authors: Alejandro Castañeda Garcia, Jan van Gemert, Daan Brinks, Nergis Tömen
• Subjects: cs.CV, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2410.01376
• Pdf URL: https://arxiv.org/pdf/2410.01376
• Copy Paste: [[2410.01376]] Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systems(https://arxiv.org/abs/2410.01376)
• Keywords: robust
• Abstract: Extracting physical dynamical system parameters from videos is of great interest to applications in natural science and technology. The state-of-the-art in automatic parameter estimation from video is addressed by training supervised deep networks on large datasets. Such datasets require labels, which are difficult to acquire. While some unsupervised techniques -- which depend on frame prediction -- exist, they suffer from long training times, instability under different initializations, and are limited to hand-picked motion problems. In this work, we propose a method to estimate the physical parameters of any known, continuous governing equation from single videos; our solution is suitable for different dynamical systems beyond motion and is robust to initialization compared to previous approaches. Moreover, we remove the need for frame prediction by implementing a KL-divergence-based loss function in the latent space, which avoids convergence to trivial solutions and reduces model size and compute.

Title: FLAME: Adaptive and Reactive Concept Drift Mitigation for Federated Learning Deployments

• Authors: Ioannis Mavromatis, Stefano De Feo, Aftab Khan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01386
• Pdf URL: https://arxiv.org/pdf/2410.01386
• Copy Paste: [[2410.01386]] FLAME: Adaptive and Reactive Concept Drift Mitigation for Federated Learning Deployments(https://arxiv.org/abs/2410.01386)
• Keywords: privacy, robust, federate
• Abstract: This paper presents Federated Learning with Adaptive Monitoring and Elimination (FLAME), a novel solution capable of detecting and mitigating concept drift in Federated Learning (FL) Internet of Things (IoT) environments. Concept drift poses significant challenges for FL models deployed in dynamic and real-world settings. FLAME leverages an FL architecture, considers a real-world FL pipeline, and proves capable of maintaining model performance and accuracy while addressing bandwidth and privacy constraints. Introducing various features and extensions on previous works, FLAME offers a robust solution to concept drift, significantly reducing computational load and communication overhead. Compared to well-known lightweight mitigation methods, FLAME demonstrates superior performance in maintaining high F1 scores and reducing resource utilisation in large-scale IoT deployments, making it a promising approach for real-world applications.

Title: Causal Inference Tools for a Better Evaluation of Machine Learning

• Authors: Michaël Soumm
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01392
• Pdf URL: https://arxiv.org/pdf/2410.01392
• Copy Paste: [[2410.01392]] Causal Inference Tools for a Better Evaluation of Machine Learning(https://arxiv.org/abs/2410.01392)
• Keywords: robust, fair
• Abstract: We present a comprehensive framework for applying rigorous statistical techniques from econometrics to analyze and improve machine learning systems. We introduce key statistical methods such as Ordinary Least Squares (OLS) regression, Analysis of Variance (ANOVA), and logistic regression, explaining their theoretical foundations and practical applications in machine learning evaluation. The document serves as a guide for researchers and practitioners, detailing how these techniques can provide deeper insights into model behavior, performance, and fairness. We cover the mathematical principles behind each method, discuss their assumptions and limitations, and provide step-by-step instructions for their implementation. The paper also addresses how to interpret results, emphasizing the importance of statistical significance and effect size. Through illustrative examples, we demonstrate how these tools can reveal subtle patterns and interactions in machine learning models that are not apparent from traditional evaluation metrics. By connecting the fields of econometrics and machine learning, this work aims to equip readers with powerful analytical tools for more rigorous and comprehensive evaluation of AI systems. The framework presented here contributes to developing more robust, interpretable, and fair machine learning technologies.

Title: Signal Adversarial Examples Generation for Signal Detection Network via White-Box Attack

• Authors: Dongyang Li, Linyuan Wang, Guangwei Xiong, Bin Yan, Dekui Ma, Jinxian Peng
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.01393
• Pdf URL: https://arxiv.org/pdf/2410.01393
• Copy Paste: [[2410.01393]] Signal Adversarial Examples Generation for Signal Detection Network via White-Box Attack(https://arxiv.org/abs/2410.01393)
• Keywords: security, attack
• Abstract: With the development and application of deep learning in signal detection tasks, the vulnerability of neural networks to adversarial attacks has also become a security threat to signal detection networks. This paper defines a signal adversarial examples generation model for signal detection network from the perspective of adding perturbations to the signal. The model uses the inequality relationship of L2-norm between time domain and time-frequency domain to constrain the energy of signal perturbations. Building upon this model, we propose a method for generating signal adversarial examples utilizing gradient-based attacks and Short-Time Fourier Transform. The experimental results show that under the constraint of signal perturbation energy ratio less than 3%, our adversarial attack resulted in a 28.1% reduction in the mean Average Precision (mAP), a 24.7% reduction in recall, and a 30.4% reduction in precision of the signal detection network. Compared to random noise perturbation of equivalent intensity, our adversarial attack demonstrates a significant attack effect.

Title: CrowdCounter: A benchmark type-specific multi-target counterspeech dataset

• Authors: Punyajoy Saha, Abhilash Datta, Abhik Jana, Animesh Mukherjee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01400
• Pdf URL: https://arxiv.org/pdf/2410.01400
• Copy Paste: [[2410.01400]] CrowdCounter: A benchmark type-specific multi-target counterspeech dataset(https://arxiv.org/abs/2410.01400)
• Keywords: large language model
• Abstract: Counterspeech presents a viable alternative to banning or suspending users for hate speech while upholding freedom of expression. However, writing effective counterspeech is challenging for moderators/users. Hence, developing suggestion tools for writing counterspeech is the need of the hour. One critical challenge in developing such a tool is the lack of quality and diversity of the responses in the existing datasets. Hence, we introduce a new dataset - CrowdCounter containing 3,425 hate speech-counterspeech pairs spanning six different counterspeech types (empathy, humor, questioning, warning, shaming, contradiction), which is the first of its kind. The design of our annotation platform itself encourages annotators to write type-specific, non-redundant and high-quality counterspeech. We evaluate two frameworks for generating counterspeech responses - vanilla and type-controlled prompts - across four large language models. In terms of metrics, we evaluate the responses using relevance, diversity and quality. We observe that Flan-T5 is the best model in the vanilla framework across different models. Type-specific prompts enhance the relevance of the responses, although they might reduce the language quality. DialoGPT proves to be the best at following the instructions and generating the type-specific counterspeech accurately.

Title: Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answering

• Authors: Yu Zhang, Kehai Chen, Xuefeng Bai, zhao kang, Quanjiang Guo, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01401
• Pdf URL: https://arxiv.org/pdf/2410.01401
• Copy Paste: [[2410.01401]] Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answering(https://arxiv.org/abs/2410.01401)
• Keywords: large language model
• Abstract: Knowledge graph question answering (KGQA) involves answering natural language questions by leveraging structured information stored in a knowledge graph. Typically, KGQA initially retrieve a targeted subgraph from a large-scale knowledge graph, which serves as the basis for reasoning models to address queries. However, the retrieved subgraph inevitably brings distraction information for knowledge utilization, impeding the model's ability to perform accurate reasoning. To address this issue, we propose a Question-guided Knowledge Graph Re-scoring method (Q-KGR) to eliminate noisy pathways for the input question, thereby focusing specifically on pertinent factual knowledge. Moreover, we introduce Knowformer, a parameter-efficient method for injecting the re-scored knowledge graph into large language models to enhance their ability to perform factual reasoning. Extensive experiments on multiple KGQA benchmarks demonstrate the superiority of our method over existing systems.

Title: On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding

• Authors: Kevin Xu, Issei Sato
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01405
• Pdf URL: https://arxiv.org/pdf/2410.01405
• Copy Paste: [[2410.01405]] On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding(https://arxiv.org/abs/2410.01405)
• Keywords: transformer
• Abstract: Looped Transformers offer advantages in parameter efficiency and Turing completeness. However, their expressive power for function approximation and approximation rate remains underexplored. In this paper, we establish approximation rates of Looped Transformers by defining the concept of the modulus of continuity for sequence-to-sequence functions. This reveals a limitation specific to the looped architecture. That is, the analysis prompts us to incorporate scaling parameters for each loop, conditioned on timestep encoding. Experimental results demonstrate that increasing the number of loops enhances performance, with further gains achieved through the timestep encoding architecture.

Title: The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs

• Authors: Hong Li, Nanxi Li, Yuanjie Chen, Jianbin Zhu, Qinlu Guo, Cewu Lu, Yong-Lu Li
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01417
• Pdf URL: https://arxiv.org/pdf/2410.01417
• Copy Paste: [[2410.01417]] The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs(https://arxiv.org/abs/2410.01417)
• Keywords: large language model
• Abstract: Multi-modal Large Language Models (MLLMs) have exhibited impressive capability. However, recently many deficiencies of MLLMs have been found compared to human intelligence, $\textit{e.g.}$, hallucination. To drive the MLLMs study, the community dedicated efforts to building larger benchmarks with complex tasks. In this paper, we propose benchmarking an essential but usually overlooked intelligence: $\textbf{association}$, a human's basic capability to link observation and prior practice memory. To comprehensively investigate MLLM's performance on the association, we formulate the association task and devise a standard benchmark based on adjective and verb semantic concepts. Instead of costly data annotation and curation, we propose a convenient $\textbf{annotation-free}$ construction method transforming the general dataset for our association tasks. Simultaneously, we devise a rigorous data refinement process to eliminate confusion in the raw dataset. Building on this database, we establish three levels of association tasks: single-step, synchronous, and asynchronous associations. Moreover, we conduct a comprehensive investigation into the MLLMs' zero-shot association capabilities, addressing multiple dimensions, including three distinct memory strategies, both open-source and closed-source MLLMs, cutting-edge Mixture-of-Experts (MoE) models, and the involvement of human experts. Our systematic investigation shows that current open-source MLLMs consistently exhibit poor capability in our association tasks, even the currently state-of-the-art GPT-4V(vision) also has a significant gap compared to humans. We believe our benchmark would pave the way for future MLLM studies. $\textit{Our data and code are available at:}$ this https URL.

Title: Fair4Free: Generating High-fidelity Fair Synthetic Samples using Data Free Distillation

• Authors: Md Fahim Sikder, Daniel de Leng, Fredrik Heintz
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01423
• Pdf URL: https://arxiv.org/pdf/2410.01423
• Copy Paste: [[2410.01423]] Fair4Free: Generating High-fidelity Fair Synthetic Samples using Data Free Distillation(https://arxiv.org/abs/2410.01423)
• Keywords: fair, data-free, generative
• Abstract: This work presents Fair4Free, a novel generative model to generate synthetic fair data using data-free distillation in the latent space. Fair4Free can work on the situation when the data is private or inaccessible. In our approach, we first train a teacher model to create fair representation and then distil the knowledge to a student model (using a smaller architecture). The process of distilling the student model is data-free, i.e. the student model does not have access to the training dataset while distilling. After the distillation, we use the distilled model to generate fair synthetic samples. Our extensive experiments show that our synthetic samples outperform state-of-the-art models in all three criteria (fairness, utility and synthetic quality) with a performance increase of 5% for fairness, 8% for utility and 12% in synthetic quality for both tabular and image datasets.

Title: Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks

• Authors: Xingxuan Li, Weiwen Xu, Ruochen Zhao, Fangkai Jiao, Shafiq Joty, Lidong Bing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01428
• Pdf URL: https://arxiv.org/pdf/2410.01428
• Copy Paste: [[2410.01428]] Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks(https://arxiv.org/abs/2410.01428)
• Keywords: large language model
• Abstract: State-of-the-art large language models (LLMs) exhibit impressive problem-solving capabilities but may struggle with complex reasoning and factual correctness. Existing methods harness the strengths of chain-of-thought and retrieval-augmented generation (RAG) to decompose a complex problem into simpler steps and apply retrieval to improve factual correctness. These methods work well on straightforward reasoning tasks but often falter on challenging tasks such as competitive programming and mathematics, due to frequent reasoning errors and irrelevant knowledge retrieval. To address this, we introduce Critic-guided planning with Retrieval-augmentation, CR-Planner, a novel framework that leverages fine-tuned critic models to guide both reasoning and retrieval processes through planning. CR-Planner solves a problem by iteratively selecting and executing sub-goals. Initially, it identifies the most promising sub-goal from reasoning, query generation, and retrieval, guided by rewards given by a critic model named sub-goal critic. It then executes this sub-goal through sampling and selecting the optimal output based on evaluations from another critic model named execution critic. This iterative process, informed by retrieved information and critic models, enables CR-Planner to effectively navigate the solution space towards the final answer. We employ Monte Carlo Tree Search to collect the data for training the critic models, allowing for a systematic exploration of action sequences and their long-term impacts. We validate CR-Planner on challenging domain-knowledge-intensive and reasoning-heavy tasks, including competitive programming, theorem-driven math reasoning, and complex domain retrieval problems. Our experiments demonstrate that CR-Planner significantly outperforms baselines, highlighting its effectiveness in addressing challenging problems by improving both reasoning and retrieval.

Title: Scalable Reinforcement Learning-based Neural Architecture Search

• Authors: Amber Cassimon, Siegfried Mercelis, Kevin Mets
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01431
• Pdf URL: https://arxiv.org/pdf/2410.01431
• Copy Paste: [[2410.01431]] Scalable Reinforcement Learning-based Neural Architecture Search(https://arxiv.org/abs/2410.01431)
• Keywords: robust
• Abstract: In this publication, we assess the ability of a novel Reinforcement Learning-based solution to the problem of Neural Architecture Search, where a Reinforcement Learning (RL) agent learns to search for good architectures, rather than to return a single optimal architecture. We consider both the NAS-Bench-101 and NAS- Bench-301 settings, and compare against various known strong baselines, such as local search and random search. We conclude that our Reinforcement Learning agent displays strong scalability with regards to the size of the search space, but limited robustness to hyperparameter changes.

Title: Adaptive teachers for amortized samplers

• Authors: Minsu Kim, Sanghyeok Choi, Taeyoung Yun, Emmanuel Bengio, Leo Feng, Jarrid Rector-Brooks, Sungsoo Ahn, Jinkyoo Park, Nikolay Malkin, Yoshua Bengio
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01432
• Pdf URL: https://arxiv.org/pdf/2410.01432
• Copy Paste: [[2410.01432]] Adaptive teachers for amortized samplers(https://arxiv.org/abs/2410.01432)
• Keywords: diffusion, generative
• Abstract: Amortized inference is the task of training a parametric model, such as a neural network, to approximate a distribution with a given unnormalized density where exact sampling is intractable. When sampling is implemented as a sequential decision-making process, reinforcement learning (RL) methods, such as generative flow networks, can be used to train the sampling policy. Off-policy RL training facilitates the discovery of diverse, high-reward candidates, but existing methods still face challenges in efficient exploration. We propose to use an adaptive training distribution (the Teacher) to guide the training of the primary amortized sampler (the Student) by prioritizing high-loss regions. The Teacher, an auxiliary behavior model, is trained to sample high-error regions of the Student and can generalize across unexplored modes, thereby enhancing mode coverage by providing an efficient training curriculum. We validate the effectiveness of this approach in a synthetic environment designed to present an exploration challenge, two diffusion-based sampling tasks, and four biochemical discovery tasks demonstrating its ability to improve sample efficiency and mode coverage.

Title: Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models

• Authors: Philipp Mondorf, Sondre Wold, Barbara Plank
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01434
• Pdf URL: https://arxiv.org/pdf/2410.01434
• Copy Paste: [[2410.01434]] Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models(https://arxiv.org/abs/2410.01434)
• Keywords: interpretability, transformer
• Abstract: A fundamental question in interpretability research is to what extent neural networks, particularly language models, implement reusable functions via subnetworks that can be composed to perform more complex tasks. Recent developments in mechanistic interpretability have made progress in identifying subnetworks, often referred to as circuits, which represent the minimal computational subgraph responsible for a model's behavior on specific tasks. However, most studies focus on identifying circuits for individual tasks without investigating how functionally similar circuits relate to each other. To address this gap, we examine the modularity of neural networks by analyzing circuits for highly compositional subtasks within a transformer-based language model. Specifically, given a probabilistic context-free grammar, we identify and compare circuits responsible for ten modular string-edit operations. Our results indicate that functionally similar circuits exhibit both notable node overlap and cross-task faithfulness. Moreover, we demonstrate that the circuits identified can be reused and combined through subnetwork set operations to represent more complex functional capabilities of the model.

Title: Information-Theoretical Principled Trade-off between Jailbreakability and Stealthiness on Vision Language Models

• Authors: Ching-Chia Kao, Chia-Mu Yu, Chun-Shien Lu, Chu-Song Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01438
• Pdf URL: https://arxiv.org/pdf/2410.01438
• Copy Paste: [[2410.01438]] Information-Theoretical Principled Trade-off between Jailbreakability and Stealthiness on Vision Language Models(https://arxiv.org/abs/2410.01438)
• Keywords: attack, robust, steal, diffusion
• Abstract: In recent years, Vision-Language Models (VLMs) have demonstrated significant advancements in artificial intelligence, transforming tasks across various domains. Despite their capabilities, these models are susceptible to jailbreak attacks, which can compromise their safety and reliability. This paper explores the trade-off between jailbreakability and stealthiness in VLMs, presenting a novel algorithm to detect non-stealthy jailbreak attacks and enhance model robustness. We introduce a stealthiness-aware jailbreak attack using diffusion models, highlighting the challenge of detecting AI-generated content. Our approach leverages Fano's inequality to elucidate the relationship between attack success rates and stealthiness scores, providing an explainable framework for evaluating these threats. Our contributions aim to fortify AI systems against sophisticated attacks, ensuring their outputs remain aligned with ethical standards and user expectations.

Title: Agent-Driven Large Language Models for Mandarin Lyric Generation

• Authors: Hong-Hsiang Liu, Yi-Wen Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01450
• Pdf URL: https://arxiv.org/pdf/2410.01450
• Copy Paste: [[2410.01450]] Agent-Driven Large Language Models for Mandarin Lyric Generation(https://arxiv.org/abs/2410.01450)
• Keywords: diffusion, generative, large language model
• Abstract: Generative Large Language Models have shown impressive in-context learning abilities, performing well across various tasks with just a prompt. Previous melody-to-lyric research has been limited by scarce high-quality aligned data and unclear standard for creativeness. Most efforts focused on general themes or emotions, which are less valuable given current language model capabilities. In tonal contour languages like Mandarin, pitch contours are influenced by both melody and tone, leading to variations in lyric-melody fit. Our study, validated by the Mpop600 dataset, confirms that lyricists and melody writers consider this fit during their composition process. In this research, we developed a multi-agent system that decomposes the melody-to-lyric task into sub-tasks, with each agent controlling rhyme, syllable count, lyric-melody alignment, and consistency. Listening tests were conducted via a diffusion-based singing voice synthesizer to evaluate the quality of lyrics generated by different agent groups.

Title: Verbalized Graph Representation Learning: A Fully Interpretable Graph Model Based on Large Language Models Throughout the Entire Process

• Authors: Xingyu Ji, Jiale Liu, Lu Li, Maojun Wang, Zeyu Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01457
• Pdf URL: https://arxiv.org/pdf/2410.01457
• Copy Paste: [[2410.01457]] Verbalized Graph Representation Learning: A Fully Interpretable Graph Model Based on Large Language Models Throughout the Entire Process(https://arxiv.org/abs/2410.01457)
• Keywords: interpretability, large language model
• Abstract: Representation learning on text-attributed graphs (TAGs) has attracted significant interest due to its wide-ranging real-world applications, particularly through Graph Neural Networks (GNNs). Traditional GNN methods focus on encoding the structural information of graphs, often using shallow text embeddings for node or edge attributes. This limits the model to understand the rich semantic information in the data and its reasoning ability for complex downstream tasks, while also lacking interpretability. With the rise of large language models (LLMs), an increasing number of studies are combining them with GNNs for graph representation learning and downstream tasks. While these approaches effectively leverage the rich semantic information in TAGs datasets, their main drawback is that they are only partially interpretable, which limits their application in critical fields. In this paper, we propose a verbalized graph representation learning (VGRL) method which is fully interpretable. In contrast to traditional graph machine learning models, which are usually optimized within a continuous parameter space, VGRL constrains this parameter space to be text description which ensures complete interpretability throughout the entire process, making it easier for users to understand and trust the decisions of the model. We conduct several studies to empirically evaluate the effectiveness of VGRL and we believe these method can serve as a stepping stone in graph representation learning.

Title: Selective Aggregation for Low-Rank Adaptation in Federated Learning

• Authors: Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, Liangqiong Qu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01463
• Pdf URL: https://arxiv.org/pdf/2410.01463
• Copy Paste: [[2410.01463]] Selective Aggregation for Low-Rank Adaptation in Federated Learning(https://arxiv.org/abs/2410.01463)
• Keywords: federate
• Abstract: We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned $A$ and $B$ matrices. In doing so, we uncover that $A$ matrices are responsible for learning general knowledge, while $B$ matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices $A$ and $B$ to model the weight update, but only $A$ matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned $A$ and $B$ matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method.

Title: SinkSAM: A Monocular Depth-Guided SAM Framework for Automatic Sinkhole Segmentation

• Authors: Osher Rafaeli, Tal Svoray, Ariel Nahlieli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01473
• Pdf URL: https://arxiv.org/pdf/2410.01473
• Copy Paste: [[2410.01473]] SinkSAM: A Monocular Depth-Guided SAM Framework for Automatic Sinkhole Segmentation(https://arxiv.org/abs/2410.01473)
• Keywords: robust, segmentation
• Abstract: Soil sinkholes significantly influence soil degradation, but their irregular shapes, along with interference from shadow and vegetation, make it challenging to accurately quantify their properties using remotely sensed data. We present a novel framework for sinkhole segmentation that combines traditional topographic computations of closed depressions with the newly developed prompt-based Segment Anything Model (SAM). Within this framework, termed SinkSAM, we highlight four key improvements: (1) The integration of topographic computations with SAM enables pixel-level refinement of sinkhole boundaries segmentation; (2) A coherent mathematical prompting strategy, based on closed depressions, addresses the limitations of purely learning-based models (CNNs) in detecting and segmenting undefined sinkhole features, while improving generalization to new, unseen regions; (3) Using Depth Anything V2 monocular depth for automatic prompts eliminates photogrammetric biases, enabling sinkhole mapping without the dependence on LiDAR data; and (4) An established sinkhole database facilitates fine-tuning of SAM, improving its zero-shot performance in sinkhole segmentation. These advancements allow the deployment of SinkSAM, in an unseen test area, in the highly variable semiarid region, achieving an intersection-over-union (IoU) of 40.27\% and surpassing previous results. This paper also presents the first SAM implementation for sinkhole segmentation and demonstrates the robustness of SinkSAM in extracting sinkhole maps using a single RGB image.

Title: Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks

• Authors: Edan Kinderman, Itay Hubara, Haggai Maron, Daniel Soudry
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01483
• Pdf URL: https://arxiv.org/pdf/2410.01483
• Copy Paste: [[2410.01483]] Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks(https://arxiv.org/abs/2410.01483)
• Keywords: transformer
• Abstract: Many recent methods aim to merge neural networks (NNs) with identical architectures trained on different tasks to obtain a single multi-task model. Most existing works tackle the simpler setup of merging NNs initialized from a common pre-trained network, where simple heuristics like weight averaging work well. This work targets a more challenging goal: merging large transformers trained on different tasks from distinct initializations. First, we demonstrate that traditional merging methods fail catastrophically in this setup. To overcome this challenge, we propose Foldable SuperNet Merge (FS-Merge), a method that optimizes a SuperNet to fuse the original models using a feature reconstruction loss. FS-Merge is simple, data-efficient, and capable of merging models of varying widths. We test FS-Merge against existing methods, including knowledge distillation, on MLPs and transformers across various settings, sizes, tasks, and modalities. FS-Merge consistently outperforms them, achieving SOTA results, particularly in limited data scenarios.

Title: A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts

• Authors: Suyu Ge, Xihui Lin, Yunan Zhang, Jiawei Han, Hao Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01485
• Pdf URL: https://arxiv.org/pdf/2410.01485
• Copy Paste: [[2410.01485]] A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts(https://arxiv.org/abs/2410.01485)
• Keywords: large language model
• Abstract: Training and serving long-context large language models (LLMs) incurs substantial overhead. To address this, two critical steps are often required: a pretrained LLM typically undergoes a separate stage for context length extension by training on long-context data, followed by architectural modifications to reduce the overhead of KV cache during serving. This paper argues that integrating length extension with a GPU-friendly KV cache reduction architecture not only reduces training overhead during length extension, but also achieves better long-context performance. This leads to our proposed LongGen, which finetunes a pretrained LLM into an efficient architecture during length extension. LongGen builds on three key insights: (1) Sparse attention patterns, such as window attention (attending to recent tokens), attention sink (initial ones), and blockwise sparse attention (strided token blocks) are well-suited for building efficient long-context models, primarily due to their GPU-friendly memory access patterns, enabling efficiency gains not just theoretically but in practice as well. (2) It is essential for the model to have direct access to all tokens. A hybrid architecture with 1/3 full attention layers and 2/3 efficient ones achieves a balanced trade-off between efficiency and long-context performance. (3) Lightweight training on 5B long-context data is sufficient to extend the hybrid model's context length from 4K to 128K. We evaluate LongGen on both Llama-2 7B and Llama-2 70B, demonstrating its effectiveness across different scales. During training with 128K-long contexts, LongGen achieves 1.55x training speedup and reduces wall-clock time by 36%, compared to a full-attention baseline. During inference, LongGen reduces KV cache memory by 62%, achieving 1.67x prefilling speedup and 1.41x decoding speedup.

Title: Extending Context Window of Large Language Models from a Distributional Perspective

• Authors: Yingsheng Wu. Yuxuan Gu, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01490
• Pdf URL: https://arxiv.org/pdf/2410.01490
• Copy Paste: [[2410.01490]] Extending Context Window of Large Language Models from a Distributional Perspective(https://arxiv.org/abs/2410.01490)
• Keywords: large language model
• Abstract: Scaling the rotary position embedding (RoPE) has become a common method for extending the context window of RoPE-based large language models (LLMs). However, existing scaling methods often rely on empirical approaches and lack a profound understanding of the internal distribution within RoPE, resulting in suboptimal performance in extending the context window length. In this paper, we propose to optimize the context window extending task from the view of rotary angle distribution. Specifically, we first estimate the distribution of the rotary angles within the model and analyze the extent to which length extension perturbs this distribution. Then, we present a novel extension strategy that minimizes the disturbance between rotary angle distributions to maintain consistency with the pre-training phase, enhancing the model's capability to generalize to longer sequences. Experimental results compared to the strong baseline methods demonstrate that our approach reduces by up to 72% of the distributional disturbance when extending LLaMA2's context window to 8k, and reduces by up to 32% when extending to 16k. On the LongBench-E benchmark, our method achieves an average improvement of up to 4.33% over existing state-of-the-art methods. Furthermore, Our method maintains the model's performance on the Hugging Face Open LLM benchmark after context window extension, with only an average performance fluctuation ranging from -0.12 to +0.22.

Title: DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

• Authors: Yuxuan Zhang, Ruizhe Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01497
• Pdf URL: https://arxiv.org/pdf/2410.01497
• Copy Paste: [[2410.01497]] DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models(https://arxiv.org/abs/2410.01497)
• Keywords: robust, large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have achieved robust performance across diverse tasks, but fine-tuning these models for specific domains remains resource-intensive. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) address this challenge by fine-tuning a small subset of parameters. However, existing methods for fusing multiple LoRAs lack dynamic fusion based on contextual inputs and often increase inference time due to token-level operations. We propose DLP-LoRA, a Dynamic Lightweight Plugin that employs a mini-MLP module with only 5M parameters to dynamically fuse multiple LoRAs at the sentence level using top-p sampling strategies. This approach reduces inference time to less than twice that of single LoRA inference by leveraging parallel computation. Evaluations across 26 tasks-including multiple-choice questions and question answering-demonstrate that DLP-LoRA achieves an average accuracy of 92.34% on multiple-choice datasets and significant improvements in BLEU and ROUGE scores on QA datasets, outperforming different LLMs backbones under composite task settings. DLP-LoRA effectively balances performance and efficiency, making it a practical solution for dynamic multi-task adaptation in LLMs. Our code is available at this https URL.

Title: Discrete Diffusion Schr\"odinger Bridge Matching for Graph Transformation

• Authors: Jun Hyeong Kim, Seonghwan Kim, Seokhyun Moon, Hyeongwoo Kim, Jeheon Woo, Woo Youn Kim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01500
• Pdf URL: https://arxiv.org/pdf/2410.01500
• Copy Paste: [[2410.01500]] Discrete Diffusion Schr\"odinger Bridge Matching for Graph Transformation(https://arxiv.org/abs/2410.01500)
• Keywords: diffusion, generative
• Abstract: Transporting between arbitrary distributions is a fundamental goal in generative modeling. Recently proposed diffusion bridge models provide a potential solution, but they rely on a joint distribution that is difficult to obtain in practice. Furthermore, formulations based on continuous domains limit their applicability to discrete domains such as graphs. To overcome these limitations, we propose Discrete Diffusion Schrödinger Bridge Matching (DDSBM), a novel framework that utilizes continuous-time Markov chains to solve the SB problem in a high-dimensional discrete state space. Our approach extends Iterative Markovian Fitting to discrete domains, and we have proved its convergence to the SB. Furthermore, we adapt our framework for the graph transformation and show that our design choice of underlying dynamics characterized by independent modifications of nodes and edges can be interpreted as the entropy-regularized version of optimal transport with a cost function described by the graph edit distance. To demonstrate the effectiveness of our framework, we have applied DDSBM to molecular optimization in the field of chemistry. Experimental results demonstrate that DDSBM effectively optimizes molecules' property-of-interest with minimal graph transformation, successfully retaining other features.

Title: PersonaMath: Enhancing Math Reasoning through Persona-Driven Data Augmentation

• Authors: Jing Luo, Run Luo, Longze Chen, Liang Zhu, Chang Ao, Jiaming Li, Yukun Chen, Xin Cheng, Wen Yang, Jiayuan Su, Chengming Li, Min Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01504
• Pdf URL: https://arxiv.org/pdf/2410.01504
• Copy Paste: [[2410.01504]] PersonaMath: Enhancing Math Reasoning through Persona-Driven Data Augmentation(https://arxiv.org/abs/2410.01504)
• Keywords: large language model
• Abstract: While closed-source Large Language Models (LLMs) demonstrate strong mathematical problem-solving abilities, open-source models continue to struggle with such tasks. To bridge this gap, we propose a data augmentation approach and introduce PersonaMathQA, a dataset derived from MATH and GSM8K, on which we train the PersonaMath models. Our approach consists of two stages: the first stage is learning from Persona Diversification, and the second stage is learning from Reflection. In the first stage, we regenerate detailed chain-of-thought (CoT) solutions as instructions using a closed-source LLM and introduce a novel persona-driven data augmentation technique to enhance the dataset's quantity and diversity. In the second stage, we incorporate reflection to fully leverage more challenging and valuable questions. Evaluation of our PersonaMath models on MATH and GSM8K reveals that the PersonaMath-7B model (based on LLaMA-2-7B) achieves an accuracy of 24.2% on MATH and 68.7% on GSM8K, surpassing all baseline methods and achieving state-of-the-art performance. Notably, our dataset contains only 70.3K data points-merely 17.8% of MetaMathQA and 27% of MathInstruct-yet our model outperforms these baselines, demonstrating the high quality and diversity of our dataset, which enables more efficient model training. We open-source the PersonaMathQA dataset, PersonaMath models, and our code for public usage.

Title: LEGO: Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion

• Authors: Dexuan Ding, Lei Wang, Liyun Zhu, Tom Gedeon, Piotr Koniusz
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01506
• Pdf URL: https://arxiv.org/pdf/2410.01506
• Copy Paste: [[2410.01506]] LEGO: Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion(https://arxiv.org/abs/2410.01506)
• Keywords: robust
• Abstract: In computer vision tasks, features often come from diverse representations, domains, and modalities, such as text, images, and videos. Effectively fusing these features is essential for robust performance, especially with the availability of powerful pre-trained models like vision-language models. However, common fusion methods, such as concatenation, element-wise operations, and non-linear techniques, often fail to capture structural relationships, deep feature interactions, and suffer from inefficiency or misalignment of features across domains. In this paper, we shift from high-dimensional feature space to a lower-dimensional, interpretable graph space by constructing similarity graphs that encode feature relationships at different levels, e.g., clip, frame, patch, token, etc. To capture deeper interactions, we use graph power expansions and introduce a learnable graph fusion operator to combine these graph powers for more effective fusion. Our approach is relationship-centric, operates in a homogeneous space, and is mathematically principled, resembling element-wise similarity score aggregation via multilinear polynomials. We demonstrate the effectiveness of our graph-based fusion method on video anomaly detection, showing strong performance across multi-representational, multi-modal, and multi-domain feature fusion tasks.

Title: Disentangling Latent Shifts of In-Context Learning Through Self-Training

• Authors: Josip Jukić, Jan Šnajder
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01508
• Pdf URL: https://arxiv.org/pdf/2410.01508
• Copy Paste: [[2410.01508]] Disentangling Latent Shifts of In-Context Learning Through Self-Training(https://arxiv.org/abs/2410.01508)
• Keywords: large language model
• Abstract: In-context learning (ICL) has become essential in natural language processing, particularly with autoregressive large language models capable of learning from demonstrations provided within the prompt. However, ICL faces challenges with stability and long contexts, especially as the number of demonstrations grows, leading to poor generalization and inefficient inference. To address these issues, we introduce STICL (Self-Training ICL), an approach that disentangles the latent shifts of demonstrations from the latent shift of the query through self-training. STICL employs a teacher model to generate pseudo-labels and trains a student model using these labels, encoded in an adapter module. The student model exhibits weak-to-strong generalization, progressively refining its predictions over time. Our empirical results show that STICL improves generalization and stability, consistently outperforming traditional ICL methods and other disentangling strategies across both in-domain and out-of-domain data.

Title: InfiniPot: Infinite Context Processing on Memory-Constrained LLMs

• Authors: Minsoo Kim, Kyuhong Shim, Jungwook Choi, Simyung Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01518
• Pdf URL: https://arxiv.org/pdf/2410.01518
• Copy Paste: [[2410.01518]] InfiniPot: Infinite Context Processing on Memory-Constrained LLMs(https://arxiv.org/abs/2410.01518)
• Keywords: large language model
• Abstract: Handling long input contexts remains a significant challenge for Large Language Models (LLMs), particularly in resource-constrained environments such as mobile devices. Our work aims to address this limitation by introducing InfiniPot, a novel KV cache control framework designed to enable pre-trained LLMs to manage extensive sequences within fixed memory constraints efficiently, without requiring additional training. InfiniPot leverages Continual Context Distillation (CCD), an iterative process that compresses and retains essential information through novel importance metrics, effectively maintaining critical data even without access to future context. Our comprehensive evaluations indicate that InfiniPot significantly outperforms models trained for long contexts in various NLP tasks, establishing its efficacy and versatility. This work represents a substantial advancement toward making LLMs applicable to a broader range of real-world scenarios.

Title: HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models

• Authors: Seanie Lee, Haebin Seong, Dong Bok Lee, Minki Kang, Xiaoyin Chen, Dominik Wagner, Yoshua Bengio, Juho Lee, Sung Ju Hwang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01524
• Pdf URL: https://arxiv.org/pdf/2410.01524
• Copy Paste: [[2410.01524]] HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models(https://arxiv.org/abs/2410.01524)
• Keywords: secure, large language model
• Abstract: Safety guard models that detect malicious queries aimed at large language models (LLMs) are essential for ensuring the secure and responsible deployment of LLMs in real-world applications. However, deploying existing safety guard models with billions of parameters alongside LLMs on mobile devices is impractical due to substantial memory requirements and latency. To reduce this cost, we distill a large teacher safety guard model into a smaller one using a labeled dataset of instruction-response pairs with binary harmfulness labels. Due to the limited diversity of harmful instructions in the existing labeled dataset, naively distilled models tend to underperform compared to larger models. To bridge the gap between small and large models, we propose HarmAug, a simple yet effective data augmentation method that involves jailbreaking an LLM and prompting it to generate harmful instructions. Given a prompt such as, "Make a single harmful instruction prompt that would elicit offensive content", we add an affirmative prefix (e.g., "I have an idea for a prompt:") to the LLM's response. This encourages the LLM to continue generating the rest of the response, leading to sampling harmful instructions. Another LLM generates a response to the harmful instruction, and the teacher model labels the instruction-response pair. We empirically show that our HarmAug outperforms other relevant baselines. Moreover, a 435-million-parameter safety guard model trained with HarmAug achieves an F1 score comparable to larger models with over 7 billion parameters, and even outperforms them in AUPRC, while operating at less than 25% of their computational cost.

Title: TiVaT: Joint-Axis Attention for Time Series Forecasting with Lead-Lag Dynamics

• Authors: Junwoo Ha, Hyukjae Kwon, Sungsoo Kim, Kisu Lee, Ha Young Kim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01531
• Pdf URL: https://arxiv.org/pdf/2410.01531
• Copy Paste: [[2410.01531]] TiVaT: Joint-Axis Attention for Time Series Forecasting with Lead-Lag Dynamics(https://arxiv.org/abs/2410.01531)
• Keywords: transformer
• Abstract: Multivariate time series (MTS) forecasting plays a crucial role in various real-world applications, yet simultaneously capturing both temporal and inter-variable dependencies remains a challenge. Conventional Channel-Dependent (CD) models handle these dependencies separately, limiting their ability to model complex interactions such as lead-lag dynamics. To address these limitations, we propose TiVaT (Time-Variable Transformer), a novel architecture that integrates temporal and variate dependencies through its Joint-Axis (JA) attention mechanism. TiVaT's ability to capture intricate variate-temporal dependencies, including asynchronous interactions, is further enhanced by the incorporation of Distance-aware Time-Variable (DTV) Sampling, which reduces noise and improves accuracy through a learned 2D map that focuses on key interactions. TiVaT effectively models both temporal and variate dependencies, consistently delivering strong performance across diverse datasets. Notably, it excels in capturing complex patterns within multivariate time series, enabling it to surpass or remain competitive with state-of-the-art methods. This positions TiVaT as a new benchmark in MTS forecasting, particularly in handling datasets characterized by intricate and challenging dependencies.

Title: Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models

• Authors: Angela Lopez-Cardona, Carlos Segura, Alexandros Karatzoglou, Sergi Abadal, Ioannis Arapakis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01532
• Pdf URL: https://arxiv.org/pdf/2410.01532
• Copy Paste: [[2410.01532]] Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models(https://arxiv.org/abs/2410.01532)
• Keywords: large language model
• Abstract: Advancements in Natural Language Processing (NLP), have led to the emergence of Large Language Models (LLMs) such as GPT, Llama, Claude, and Gemini, which excel across a range of tasks but require extensive fine-tuning to align their outputs with human expectations. A widely used method for achieving this alignment is Reinforcement Learning from Human Feedback (RLHF), which, despite its success, faces challenges in accurately modelling human preferences. In this paper, we introduce GazeReward, a novel framework that integrates implicit feedback -- and specifically eye-tracking (ET) data -- into the Reward Model (RM). In addition, we explore how ET-based features can provide insights into user preferences. Through ablation studies we test our framework with different integration methods, LLMs, and ET generator models, demonstrating that our approach significantly improves the accuracy of the RM on established human preference datasets. This work advances the ongoing discussion on optimizing AI alignment with human values, exploring the potential of cognitive data for shaping future NLP research.

Title: Toward a Holistic Evaluation of Robustness in CLIP Models

• Authors: Weijie Tu, Weijian Deng, Tom Gedeon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01534
• Pdf URL: https://arxiv.org/pdf/2410.01534
• Copy Paste: [[2410.01534]] Toward a Holistic Evaluation of Robustness in CLIP Models(https://arxiv.org/abs/2410.01534)
• Keywords: robust
• Abstract: Contrastive Language-Image Pre-training (CLIP) models have shown significant potential, particularly in zero-shot classification across diverse distribution shifts. Building on existing evaluations of overall classification robustness, this work aims to provide a more comprehensive assessment of CLIP by introducing several new perspectives. First, we investigate their robustness to variations in specific visual factors. Second, we assess two critical safety objectives--confidence uncertainty and out-of-distribution detection--beyond mere classification accuracy. Third, we evaluate the finesse with which CLIP models bridge the image and text modalities. Fourth, we extend our examination to 3D awareness in CLIP models, moving beyond traditional 2D image understanding. Finally, we explore the interaction between vision and language encoders within modern large multimodal models (LMMs) that utilize CLIP as the visual backbone, focusing on how this interaction impacts classification robustness. In each aspect, we consider the impact of six factors on CLIP models: model architecture, training distribution, training set size, fine-tuning, contrastive loss, and test-time prompts. Our study uncovers several previously unknown insights into CLIP. For instance, the architecture of the visual encoder in CLIP plays a significant role in their robustness against 3D corruption. CLIP models tend to exhibit a bias towards shape when making predictions. Moreover, this bias tends to diminish after fine-tuning on ImageNet. Vision-language models like LLaVA, leveraging the CLIP vision encoder, could exhibit benefits in classification performance for challenging categories over CLIP alone. Our findings are poised to offer valuable guidance for enhancing the robustness and reliability of CLIP models.

Title: GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussians

• Authors: Shuyi Jiang, Qihao Zhao, Hossein Rahmani, De Wen Soh, Jun Liu, Na Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01535
• Pdf URL: https://arxiv.org/pdf/2410.01535
• Copy Paste: [[2410.01535]] GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussians(https://arxiv.org/abs/2410.01535)
• Keywords: interpretability
• Abstract: Recently, with the development of Neural Radiance Fields and Gaussian Splatting, 3D reconstruction techniques have achieved remarkably high fidelity. However, the latent representations learnt by these methods are highly entangled and lack interpretability. In this paper, we propose a novel part-aware compositional reconstruction method, called GaussianBlock, that enables semantically coherent and disentangled representations, allowing for precise and physical editing akin to building blocks, while simultaneously maintaining high fidelity. Our GaussianBlock introduces a hybrid representation that leverages the advantages of both primitives, known for their flexible actionability and editability, and 3D Gaussians, which excel in reconstruction quality. Specifically, we achieve semantically coherent primitives through a novel attention-guided centering loss derived from 2D semantic priors, complemented by a dynamic splitting and fusion strategy. Furthermore, we utilize 3D Gaussians that hybridize with primitives to refine structural details and enhance fidelity. Additionally, a binding inheritance strategy is employed to strengthen and maintain the connection between the two. Our reconstructed scenes are evidenced to be disentangled, compositional, and compact across diverse benchmarks, enabling seamless, direct and precise editing while maintaining high quality.

Title: Multi-Scale Fusion for Object Representation

• Authors: Rongzhen Zhao, Vivienne Wang, Juho Kannala, Joni Pajarinen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01539
• Pdf URL: https://arxiv.org/pdf/2410.01539
• Copy Paste: [[2410.01539]] Multi-Scale Fusion for Object Representation(https://arxiv.org/abs/2410.01539)
• Keywords: diffusion
• Abstract: Representing images or videos as object-level feature vectors, rather than pixel-level feature maps, facilitates advanced visual tasks. Object-Centric Learning (OCL) primarily achieves this by reconstructing the input under the guidance of Variational Autoencoder (VAE) intermediate representation to drive so-called \textit{slots} to aggregate as much object information as possible. However, existing VAE guidance does not explicitly address that objects can vary in pixel sizes while models typically excel at specific pattern scales. We propose \textit{Multi-Scale Fusion} (MSF) to enhance VAE guidance for OCL training. To ensure objects of all sizes fall within VAE's comfort zone, we adopt the \textit{image pyramid}, which produces intermediate representations at multiple scales; To foster scale-invariance/variance in object super-pixels, we devise \textit{inter}/\textit{intra-scale fusion}, which augments low-quality object super-pixels of one scale with corresponding high-quality super-pixels from another scale. On standard OCL benchmarks, our technique improves mainstream methods, including state-of-the-art diffusion-based ones. The source code is available in the supplemental material.

Title: Edge-preserving noise for diffusion models

• Authors: Jente Vandersanden, Sascha Holl, Xingchang Huang, Gurprit Singh
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01540
• Pdf URL: https://arxiv.org/pdf/2410.01540
• Copy Paste: [[2410.01540]] Edge-preserving noise for diffusion models(https://arxiv.org/abs/2410.01540)
• Keywords: robust, diffusion, generative
• Abstract: Classical generative diffusion models learn an isotropic Gaussian denoising process, treating all spatial regions uniformly, thus neglecting potentially valuable structural information in the data. Inspired by the long-established work on anisotropic diffusion in image processing, we present a novel edge-preserving diffusion model that is a generalization of denoising diffusion probablistic models (DDPM). In particular, we introduce an edge-aware noise scheduler that varies between edge-preserving and isotropic Gaussian noise. We show that our model's generative process converges faster to results that more closely match the target distribution. We demonstrate its capability to better learn the low-to-mid frequencies within the dataset, which plays a crucial role in representing shapes and structural information. Our edge-preserving diffusion process consistently outperforms state-of-the-art baselines in unconditional image generation. It is also more robust for generative tasks guided by a shape-based prior, such as stroke-to-image generation. We present qualitative and quantitative results showing consistent improvements (FID score) of up to 30% for both tasks.

Title: Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension

• Authors: Zaiquan Yang, Yuhao Liu, Jiaying Lin, Gerhard Hancke, Rynson W.H. Lau
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01544
• Pdf URL: https://arxiv.org/pdf/2410.01544
• Copy Paste: [[2410.01544]] Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension(https://arxiv.org/abs/2410.01544)
• Keywords: large language model, segmentation
• Abstract: This paper explores the weakly-supervised referring image segmentation (WRIS) problem, and focuses on a challenging setup where target localization is learned directly from image-text pairs. We note that the input text description typically already contains detailed information on how to localize the target object, and we also observe that humans often follow a step-by-step comprehension process (\ie, progressively utilizing target-related attributes and relations as cues) to identify the target object. Hence, we propose a novel Progressive Comprehension Network (PCNet) to leverage target-related textual cues from the input description for progressively localizing the target object. Specifically, we first use a Large Language Model (LLM) to decompose the input text description into short phrases. These short phrases are taken as target-related cues and fed into a Conditional Referring Module (CRM) in multiple stages, to allow updating the referring text embedding and enhance the response map for target localization in a multi-stage manner. Based on the CRM, we then propose a Region-aware Shrinking (RaS) loss to constrain the visual localization to be conducted progressively in a coarse-to-fine manner across different stages. Finally, we introduce an Instance-aware Disambiguation (IaD) loss to suppress instance localization ambiguity by differentiating overlapping response maps generated by different referring texts on the same image. Extensive experiments show that our method outperforms SOTA methods on three common benchmarks.

Title: Lines of Thought in Large Language Models

• Authors: Raphaël Sarfati, Toni J. B. Liu, Nicolas Boullé, Christopher J. Earls
• Subjects: cs.LG, physics.data-an
• Abstract URL: https://arxiv.org/abs/2410.01545
• Pdf URL: https://arxiv.org/pdf/2410.01545
• Copy Paste: [[2410.01545]] Lines of Thought in Large Language Models(https://arxiv.org/abs/2410.01545)
• Keywords: transformer, large language model
• Abstract: Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajectories realize different contextualization, or 'thinking', steps, and fully determine the output probability distribution. We aim to characterize the statistical properties of ensembles of these 'lines of thought.' We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications.

Title: In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks

• Authors: Dingzirui Wang, Xuangliang Zhang, Qiguang Chen, Longxu Dou, Xiao Xu, Rongyu Cao, Yingwei Ma, Qingfu Zhu, Wanxiang Che, Binhua Li, Fei Huang, Yongbin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01548
• Pdf URL: https://arxiv.org/pdf/2410.01548
• Copy Paste: [[2410.01548]] In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks(https://arxiv.org/abs/2410.01548)
• Keywords: large language model
• Abstract: In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs. However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs. To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer. First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task. Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average, demonstrating the effectiveness of our method.

Title: Integrative Decoding: Improve Factuality via Implicit Self-consistency

• Authors: Yi Cheng, Xiao Liang, Yeyun Gong, Wen Xiao, Song Wang, Yuji Zhang, Wenjun Hou, Kaishuai Xu, Wenge Liu, Wenjie Li, Jian Jiao, Qi Chen, Peng Cheng, Wayne Xiong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01556
• Pdf URL: https://arxiv.org/pdf/2410.01556
• Copy Paste: [[2410.01556]] Integrative Decoding: Improve Factuality via Implicit Self-consistency(https://arxiv.org/abs/2410.01556)
• Keywords: large language model
• Abstract: Self-consistency-based approaches, which involve repeatedly sampling multiple outputs and selecting the most consistent one as the final response, prove to be remarkably effective in improving the factual accuracy of large language models. Nonetheless, existing methods usually have strict constraints on the task format, largely limiting their applicability. In this paper, we present Integrative Decoding (ID), to unlock the potential of self-consistency in open-ended generation tasks. ID operates by constructing a set of inputs, each prepended with a previously sampled response, and then processes them concurrently, with the next token being selected by aggregating of all their corresponding predictions at each decoding step. In essence, this simple approach implicitly incorporates self-consistency in the decoding objective. Extensive evaluation shows that ID consistently enhances factuality over a wide range of language models, with substantial improvements on the TruthfulQA (+11.2%), Biographies (+15.4%) and LongFact (+8.5%) benchmarks. The performance gains amplify progressively as the number of sampled responses increases, indicating the potential of ID to scale up with repeated sampling.

Title: OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data

• Authors: Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, Igor Gitman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01560
• Pdf URL: https://arxiv.org/pdf/2410.01560
• Copy Paste: [[2410.01560]] OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data(https://arxiv.org/abs/2410.01560)
• Keywords: robust, large language model
• Abstract: Mathematical reasoning continues to be a critical challenge in large language model (LLM) development with significant interest. However, most of the cutting-edge progress in mathematical reasoning with LLMs has become \emph{closed-source} due to lack of access to training data. This lack of data access limits researchers from understanding the impact of different choices for synthesizing and utilizing the data. With the goal of creating a high-quality finetuning (SFT) dataset for math reasoning, we conduct careful ablation experiments on data synthesis using the recently released \texttt{Llama3.1} family of models. Our experiments show that: (a) solution format matters, with excessively verbose solutions proving detrimental to SFT performance, (b) data generated by a strong teacher outperforms \emph{on-policy} data generated by a weak student model, (c) SFT is robust to low-quality solutions, allowing for imprecise data filtering, and (d) question diversity is crucial for achieving data scaling gains. Based on these insights, we create the OpenMathInstruct-2 dataset, which consists of 14M question-solution pairs ($\approx$ 600K unique questions), making it nearly eight times larger than the previous largest open-source math reasoning dataset. Finetuning the \texttt{Llama-3.1-8B-Base} using OpenMathInstruct-2 outperforms \texttt{Llama3.1-8B-Instruct} on MATH by an absolute 15.9\% (51.9\% $\rightarrow$ 67.8\%). Finally, to accelerate the open-source efforts, we release the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license.

Title: Bayes' Power for Explaining In-Context Learning Generalizations

• Authors: Samuel Müller, Noah Hollmann, Frank Hutter
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01565
• Pdf URL: https://arxiv.org/pdf/2410.01565
• Copy Paste: [[2410.01565]] Bayes' Power for Explaining In-Context Learning Generalizations(https://arxiv.org/abs/2410.01565)
• Keywords: robust
• Abstract: Traditionally, neural network training has been primarily viewed as an approximation of maximum likelihood estimation (MLE). This interpretation originated in a time when training for multiple epochs on small datasets was common and performance was data bound; but it falls short in the era of large-scale single-epoch trainings ushered in by large self-supervised setups, like language models. In this new setup, performance is compute-bound, but data is readily available. As models became more powerful, in-context learning (ICL), i.e., learning in a single forward-pass based on the context, emerged as one of the dominant paradigms. In this paper, we argue that a more useful interpretation of neural network behavior in this era is as an approximation of the true posterior, as defined by the data-generating process. We demonstrate this interpretations' power for ICL and its usefulness to predict generalizations to previously unseen tasks. We show how models become robust in-context learners by effectively composing knowledge from their training data. We illustrate this with experiments that reveal surprising generalizations, all explicable through the exact posterior. Finally, we show the inherent constraints of the generalization capabilities of posteriors and the limitations of neural networks in approximating these posteriors.

Title: Adaptive Exploit Generation against Security Devices and Security APIs

• Authors: Robert Künnemann, Julian Biehl
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.01568
• Pdf URL: https://arxiv.org/pdf/2410.01568
• Copy Paste: [[2410.01568]] Adaptive Exploit Generation against Security Devices and Security APIs(https://arxiv.org/abs/2410.01568)
• Keywords: security, attack
• Abstract: Proof-of-concept exploits help demonstrate software vulnerability beyond doubt and communicate attacks to non-experts. But exploits can be configuration-specific, for example when in Security APIs, where keys are set up specifically for the application and enterprise the API serves. In this work, we show how to automatically derive proof-of-concept exploits against Security APIs using formal methods. We extend the popular protocol verifier ProVerif with a language-agnostic template mechanism. Employing program snippets attached to steps in the model, we can transform attack traces (which ProVerif typically finds automatically) into programs. Our method is general, flexible and convenient. We demonstrate its use for the W3C Web Cryptography API, for PKCS#11 and for the YubiHSM2, providing the first formal model of the latter.

Title: PASS:Test-Time Prompting to Adapt Styles and Semantic Shapes in Medical Image Segmentation

• Authors: Chuyan Zhang, Hao Zheng, Xin You, Yefeng Zheng, Yun Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01573
• Pdf URL: https://arxiv.org/pdf/2410.01573
• Copy Paste: [[2410.01573]] PASS:Test-Time Prompting to Adapt Styles and Semantic Shapes in Medical Image Segmentation(https://arxiv.org/abs/2410.01573)
• Keywords: segmentation
• Abstract: Test-time adaptation (TTA) has emerged as a promising paradigm to handle the domain shifts at test time for medical images from different institutions without using extra training data. However, existing TTA solutions for segmentation tasks suffer from (1) dependency on modifying the source training stage and access to source priors or (2) lack of emphasis on shape-related semantic knowledge that is crucial for segmentation this http URL research on visual prompt learning achieves source-relaxed adaptation by extended parameter space but still neglects the full utilization of semantic features, thus motivating our work on knowledge-enriched deep prompt learning. Beyond the general concern of image style shifts, we reveal that shape variability is another crucial factor causing the performance drop. To address this issue, we propose a TTA framework called PASS (Prompting to Adapt Styles and Semantic shapes), which jointly learns two types of prompts: the input-space prompt to reformulate the style of the test image to fit into the pretrained model and the semantic-aware prompts to bridge high-level shape discrepancy across domains. Instead of naively imposing a fixed prompt, we introduce an input decorator to generate the self-regulating visual prompt conditioned on the input data. To retrieve the knowledge representations and customize target-specific shape prompts for each test sample, we propose a cross-attention prompt modulator, which performs interaction between target representations and an enriched shape prompt bank. Extensive experiments demonstrate the superior performance of PASS over state-of-the-art methods on multiple medical image segmentation datasets. The code is available at this https URL.

Title: Fake It Until You Break It: On the Adversarial Robustness of AI-generated Image Detectors

• Authors: Sina Mavali, Jonas Ricker, David Pape, Yash Sharma, Asja Fischer, Lea Schoenherr
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01574
• Pdf URL: https://arxiv.org/pdf/2410.01574
• Copy Paste: [[2410.01574]] Fake It Until You Break It: On the Adversarial Robustness of AI-generated Image Detectors(https://arxiv.org/abs/2410.01574)
• Keywords: defense, attack, robust, generative
• Abstract: While generative AI (GenAI) offers countless possibilities for creative and productive tasks, artificially generated media can be misused for fraud, manipulation, scams, misinformation campaigns, and more. To mitigate the risks associated with maliciously generated media, forensic classifiers are employed to identify AI-generated content. However, current forensic classifiers are often not evaluated in practically relevant scenarios, such as the presence of an attacker or when real-world artifacts like social media degradations affect images. In this paper, we evaluate state-of-the-art AI-generated image (AIGI) detectors under different attack scenarios. We demonstrate that forensic classifiers can be effectively attacked in realistic settings, even when the attacker does not have access to the target model and post-processing occurs after the adversarial examples are created, which is standard on social media platforms. These attacks can significantly reduce detection accuracy to the extent that the risks of relying on detectors outweigh their benefits. Finally, we propose a simple defense mechanism to make CLIP-based detectors, which are currently the best-performing detectors, robust against these attacks.

Title: Spoken Grammar Assessment Using LLM

• Authors: Sunil Kumar Kopparapu, Chitralekha Bhat, Ashish Panda
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01579
• Pdf URL: https://arxiv.org/pdf/2410.01579
• Copy Paste: [[2410.01579]] Spoken Grammar Assessment Using LLM(https://arxiv.org/abs/2410.01579)
• Keywords: large language model
• Abstract: Spoken language assessment (SLA) systems restrict themselves to evaluating the pronunciation and oral fluency of a speaker by analysing the read and spontaneous spoken utterances respectively. The assessment of language grammar or vocabulary is relegated to written language assessment (WLA) systems. Most WLA systems present a set of sentences from a curated finite-size database of sentences thereby making it possible to anticipate the test questions and train oneself. In this paper, we propose a novel end-to-end SLA system to assess language grammar from spoken utterances thus making WLA systems redundant; additionally, we make the assessment largely unteachable by employing a large language model (LLM) to bring in variations in the test. We further demonstrate that a hybrid automatic speech recognition (ASR) with a custom-built language model outperforms the state-of-the-art ASR engine for spoken grammar assessment.

Title: Learning-Augmented Robust Algorithmic Recourse

• Authors: Kshitij Kayastha, Vasilis Gkatzelis, Shahin Jabbari
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01580
• Pdf URL: https://arxiv.org/pdf/2410.01580
• Copy Paste: [[2410.01580]] Learning-Augmented Robust Algorithmic Recourse(https://arxiv.org/abs/2410.01580)
• Keywords: robust
• Abstract: The widespread use of machine learning models in high-stakes domains can have a major negative impact, especially on individuals who receive undesirable outcomes. Algorithmic recourse provides such individuals with suggestions of minimum-cost improvements they can make to achieve a desirable outcome in the future. However, machine learning models often get updated over time and this can cause a recourse to become invalid (i.e., not lead to the desirable outcome). The robust recourse literature aims to choose recourses that are less sensitive, even against adversarial model changes, but this comes at a higher cost. To overcome this obstacle, we initiate the study of algorithmic recourse through the learning-augmented framework and evaluate the extent to which a designer equipped with a prediction regarding future model changes can reduce the cost of recourse when the prediction is accurate (consistency) while also limiting the cost even when the prediction is inaccurate (robustness). We propose a novel algorithm for this problem, study the robustness-consistency trade-off, and analyze how prediction accuracy affects performance.

Title: DynFrs: An Efficient Framework for Machine Unlearning in Random Forest

• Authors: Shurong Wang, Zhuoyang Shen, Xinbao Qiao, Tongning Zhang, Meng Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01588
• Pdf URL: https://arxiv.org/pdf/2410.01588
• Copy Paste: [[2410.01588]] DynFrs: An Efficient Framework for Machine Unlearning in Random Forest(https://arxiv.org/abs/2410.01588)
• Keywords: privacy
• Abstract: Random Forests are widely recognized for establishing efficacy in classification and regression tasks, standing out in various domains such as medical diagnosis, finance, and personalized recommendations. These domains, however, are inherently sensitive to privacy concerns, as personal and confidential data are involved. With increasing demand for the right to be forgotten, particularly under regulations such as GDPR and CCPA, the ability to perform machine unlearning has become crucial for Random Forests. However, insufficient attention was paid to this topic, and existing approaches face difficulties in being applied to real-world scenarios. Addressing this gap, we propose the DynFrs framework designed to enable efficient machine unlearning in Random Forests while preserving predictive accuracy. Dynfrs leverages subsampling method Occ(q) and a lazy tag strategy Lzy, and is still adaptable to any Random Forest variant. In essence, Occ(q) ensures that each sample in the training set occurs only in a proportion of trees so that the impact of deleting samples is limited, and Lzy delays the reconstruction of a tree node until necessary, thereby avoiding unnecessary modifications on tree structures. In experiments, applying Dynfrs on Extremely Randomized Trees yields substantial improvements, achieving orders of magnitude faster unlearning performance and better predictive accuracy than existing machine unlearning methods for Random Forests.

Title: MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation

• Authors: Mingzhen Sun, Weining Wang, Yanyuan Qiao, Jiahui Sun, Zihan Qin, Longteng Guo, Xinxin Zhu, Jing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01594
• Pdf URL: https://arxiv.org/pdf/2410.01594
• Copy Paste: [[2410.01594]] MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation(https://arxiv.org/abs/2410.01594)
• Keywords: diffusion
• Abstract: Sounding Video Generation (SVG) is an audio-video joint generation task challenged by high-dimensional signal spaces, distinct data formats, and different patterns of content information. To address these issues, we introduce a novel multi-modal latent diffusion model (MM-LDM) for the SVG task. We first unify the representation of audio and video data by converting them into a single or a couple of images. Then, we introduce a hierarchical multi-modal autoencoder that constructs a low-level perceptual latent space for each modality and a shared high-level semantic feature space. The former space is perceptually equivalent to the raw signal space of each modality but drastically reduces signal dimensions. The latter space serves to bridge the information gap between modalities and provides more insightful cross-modal guidance. Our proposed method achieves new state-of-the-art results with significant quality and efficiency gains. Specifically, our method achieves a comprehensive improvement on all evaluation metrics and a faster training and sampling speed on Landscape and AIST++ datasets. Moreover, we explore its performance on open-domain sounding video generation, long sounding video generation, audio continuation, video continuation, and conditional single-modal generation tasks for a comprehensive evaluation, where our MM-LDM demonstrates exciting adaptability and generalization ability.

Title: KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models

• Authors: Pouyan Navard, Amin Karimi Monsefi, Mengxi Zhou, Wei-Lun Chao, Alper Yilmaz, Rajiv Ramnath
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01595
• Pdf URL: https://arxiv.org/pdf/2410.01595
• Copy Paste: [[2410.01595]] KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models(https://arxiv.org/abs/2410.01595)
• Keywords: diffusion
• Abstract: Recent advances in diffusion models have significantly improved text-to-image (T2I) generation, but they often struggle to balance fine-grained precision with high-level control. Methods like ControlNet and T2I-Adapter excel at following sketches by seasoned artists but tend to be overly rigid, replicating unintentional flaws in sketches from novice users. Meanwhile, coarse-grained methods, such as sketch-based abstraction frameworks, offer more accessible input handling but lack the precise control needed for detailed, professional use. To address these limitations, we propose KnobGen, a dual-pathway framework that democratizes sketch-based image generation by seamlessly adapting to varying levels of sketch complexity and user skill. KnobGen uses a Coarse-Grained Controller (CGC) module for high-level semantics and a Fine-Grained Controller (FGC) module for detailed refinement. The relative strength of these two modules can be adjusted through our knob inference mechanism to align with the user's specific needs. These mechanisms ensure that KnobGen can flexibly generate images from both novice sketches and those drawn by seasoned artists. This maintains control over the final output while preserving the natural appearance of the image, as evidenced on the MultiGen-20M dataset and a newly collected sketch dataset.

Title: ENTP: Encoder-only Next Token Prediction

• Authors: Ethan Ewer, Daewon Chae, Thomas Zeng, Jinkyu Kim, Kangwook Lee
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01600
• Pdf URL: https://arxiv.org/pdf/2410.01600
• Copy Paste: [[2410.01600]] ENTP: Encoder-only Next Token Prediction(https://arxiv.org/abs/2410.01600)
• Keywords: transformer
• Abstract: Next-token prediction models have predominantly relied on decoder-only Transformers with causal attention, driven by the common belief that causal attention is essential to prevent "cheating" by masking future tokens. We challenge this widely accepted notion and argue that this design choice is about efficiency rather than necessity. While decoder-only Transformers are still a good choice for practical reasons, they are not the only viable option. In this work, we introduce Encoder-only Next Token Prediction (ENTP). We explore the differences between ENTP and decoder-only Transformers in expressive power and complexity, highlighting potential advantages of ENTP. We introduce the Triplet-Counting task and show, both theoretically and experimentally, that while ENTP can perform this task easily, a decoder-only Transformer cannot. Finally, we empirically demonstrate ENTP's superior performance across various realistic tasks, such as length generalization and in-context learning.

Title: Automated Red Teaming with GOAT: the Generative Offensive Agent Tester

• Authors: Maya Pavlova, Erik Brinkman, Krithika Iyer, Vitor Albiero, Joanna Bitton, Hailey Nguyen, Joe Li, Cristian Canton Ferrer, Ivan Evtimov, Aaron Grattafiori
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01606
• Pdf URL: https://arxiv.org/pdf/2410.01606
• Copy Paste: [[2410.01606]] Automated Red Teaming with GOAT: the Generative Offensive Agent Tester(https://arxiv.org/abs/2410.01606)
• Keywords: attack, generative, large language model
• Abstract: Red teaming assesses how large language models (LLMs) can produce content that violates norms, policies, and rules set during their safety training. However, most existing automated methods in the literature are not representative of the way humans tend to interact with AI models. Common users of AI models may not have advanced knowledge of adversarial machine learning methods or access to model internals, and they do not spend a lot of time crafting a single highly effective adversarial prompt. Instead, they are likely to make use of techniques commonly shared online and exploit the multiturn conversational nature of LLMs. While manual testing addresses this gap, it is an inefficient and often expensive process. To address these limitations, we introduce the Generative Offensive Agent Tester (GOAT), an automated agentic red teaming system that simulates plain language adversarial conversations while leveraging multiple adversarial prompting techniques to identify vulnerabilities in LLMs. We instantiate GOAT with 7 red teaming attacks by prompting a general-purpose model in a way that encourages reasoning through the choices of methods available, the current target model's response, and the next steps. Our approach is designed to be extensible and efficient, allowing human testers to focus on exploring new areas of risk while automation covers the scaled adversarial stress-testing of known risk territory. We present the design and evaluation of GOAT, demonstrating its effectiveness in identifying vulnerabilities in state-of-the-art LLMs, with an ASR@10 of 97% against Llama 3.1 and 88% against GPT-4 on the JailbreakBench dataset.

Title: Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging

• Authors: Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Yu Sun, Hua Wu, Sen Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01610
• Pdf URL: https://arxiv.org/pdf/2410.01610
• Copy Paste: [[2410.01610]] Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging(https://arxiv.org/abs/2410.01610)
• Keywords: large language model
• Abstract: Mixture-of-Experts (MoE) shines brightly in large language models (LLMs) and demonstrates outstanding performance in plentiful natural language processing tasks. However, existing methods transforming LLMs from dense to MoE face significant data requirements and typically rely on large-scale post-training. In this paper, we propose Upcycling Instruction Tuning (UpIT), a data-efficient approach for tuning a dense pre-trained model into a MoE instruction model. Specifically, we first point out that intermediate checkpoints during instruction tuning of the dense model are naturally suitable for specialized experts, and then propose an expert expansion stage to flexibly achieve models with flexible numbers of experts, where genetic algorithm and parameter merging are introduced to ensure sufficient diversity of new extended experts. To ensure that each specialized expert in the MoE model works as expected, we select a small amount of seed data that each expert excels to pre-optimize the router. Extensive experiments with various data scales and upcycling settings demonstrate the outstanding performance and data efficiency of UpIT, as well as stable improvement in expert or data scaling. Further analysis reveals the importance of ensuring expert diversity in upcycling.

Title: On Using Certified Training towards Empirical Robustness

• Authors: Alessandro De Palma, Serge Durand, Zakaria Chihani, François Terrier, Caterina Urban
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01617
• Pdf URL: https://arxiv.org/pdf/2410.01617
• Copy Paste: [[2410.01617]] On Using Certified Training towards Empirical Robustness(https://arxiv.org/abs/2410.01617)
• Keywords: defense, attack, robust
• Abstract: Adversarial training is arguably the most popular way to provide empirical robustness against specific adversarial examples. While variants based on multi-step attacks incur significant computational overhead, single-step variants are vulnerable to a failure mode known as catastrophic overfitting, which hinders their practical utility for large perturbations. A parallel line of work, certified training, has focused on producing networks amenable to formal guarantees of robustness against any possible attack. However, the wide gap between the best-performing empirical and certified defenses has severely limited the applicability of the latter. Inspired by recent developments in certified training, which rely on a combination of adversarial attacks with network over-approximations, and by the connections between local linearity and catastrophic overfitting, we present experimental evidence on the practical utility and limitations of using certified training towards empirical robustness. We show that, when tuned for the purpose, a recent certified training algorithm can prevent catastrophic overfitting on single-step attacks, and that it can bridge the gap to multi-step baselines under appropriate experimental settings. Finally, we present a novel regularizer for network over-approximations that can achieve similar effects while markedly reducing runtime.

Title: SGBA: Semantic Gaussian Mixture Model-Based LiDAR Bundle Adjustment

• Authors: Xingyu Ji, Shenghai Yuan, Jianping Li, Pengyu Yin, Haozhi Cao, Lihua Xie
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.01618
• Pdf URL: https://arxiv.org/pdf/2410.01618
• Copy Paste: [[2410.01618]] SGBA: Semantic Gaussian Mixture Model-Based LiDAR Bundle Adjustment(https://arxiv.org/abs/2410.01618)
• Keywords: robust
• Abstract: LiDAR bundle adjustment (BA) is an effective approach to reduce the drifts in pose estimation from the front-end. Existing works on LiDAR BA usually rely on predefined geometric features for landmark representation. This reliance restricts generalizability, as the system will inevitably deteriorate in environments where these specific features are absent. To address this issue, we propose SGBA, a LiDAR BA scheme that models the environment as a semantic Gaussian mixture model (GMM) without predefined feature types. This approach encodes both geometric and semantic information, offering a comprehensive and general representation adaptable to various environments. Additionally, to limit computational complexity while ensuring generalizability, we propose an adaptive semantic selection framework that selects the most informative semantic clusters for optimization by evaluating the condition number of the cost function. Lastly, we introduce a probabilistic feature association scheme that considers the entire probability density of assignments, which can manage uncertainties in measurement and initial pose estimation. We have conducted various experiments and the results demonstrate that SGBA can achieve accurate and robust pose refinement even in challenging scenarios with low-quality initial pose estimation and limited geometric features. We plan to open-source the work for the benefit of the community this https URL.

Title: LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models

• Authors: Zhenyue Qin, Yu Yin, Dylan Campbell, Xuansheng Wu, Ke Zou, Yih-Chung Tham, Ninghao Liu, Xiuzhen Zhang, Qingyu Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01620
• Pdf URL: https://arxiv.org/pdf/2410.01620
• Copy Paste: [[2410.01620]] LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models(https://arxiv.org/abs/2410.01620)
• Keywords: extraction
• Abstract: Ophthalmology relies heavily on detailed image analysis for diagnosis and treatment planning. While large vision-language models (LVLMs) have shown promise in understanding complex visual information, their performance on ophthalmology images remains underexplored. We introduce LMOD, a dataset and benchmark for evaluating LVLMs on ophthalmology images, covering anatomical understanding, diagnostic analysis, and demographic extraction. LMODincludes 21,993 images spanning optical coherence tomography, scanning laser ophthalmoscopy, eye photos, surgical scenes, and color fundus photographs. We benchmark 13 state-of-the-art LVLMs and find that they are far from perfect for comprehending ophthalmology images. Models struggle with diagnostic analysis and demographic extraction, reveal weaknesses in spatial reasoning, diagnostic analysis, handling out-of-domain queries, and safeguards for handling biomarkers of ophthalmology images.

Title: Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?

• Authors: Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, Guoren Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01623
• Pdf URL: https://arxiv.org/pdf/2410.01623
• Copy Paste: [[2410.01623]] Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?(https://arxiv.org/abs/2410.01623)
• Keywords: large language model
• Abstract: Low-rank training has emerged as a promising approach for reducing memory usage in training Large Language Models (LLMs). Previous methods either rely on decomposing weight matrices (e.g., LoRA), or seek to decompose gradient matrices (e.g., GaLore) to ensure reduced memory consumption. However, both of them constrain the training in a low-rank subspace, thus inevitably leading to sub-optimal performance. This raises a question: whether it is possible to consistently preserve the low-rank constraint for memory efficiency, while achieving full-rank training (i.e., training with full-rank gradients of full-rank weights) to avoid inferior outcomes? In this paper, we propose a new plug-and-play training framework for LLMs called Fira, as the first attempt to achieve this goal. First, we observe an interesting phenomenon during LLM training: the scaling impact of adaptive optimizers (e.g., Adam) on the gradient norm remains similar from low-rank to full-rank training. Based on this observation, we propose a norm-based scaling method, which utilizes the scaling impact of low-rank optimizers as substitutes for that of original full-rank optimizers to enable full-rank training. In this way, we can preserve the low-rank constraint in the optimizer while achieving full-rank training for better performance. Moreover, we find that there are sudden gradient rises during the optimization process, potentially causing loss spikes. To address this, we further put forward a norm-growth limiter to smooth the gradient via regulating the relative increase of gradient norms. Extensive experiments on the pre-training and fine-tuning of LLMs show that Fira outperforms both LoRA and GaLore, achieving performance that is comparable to or even better than full-rank training.

Title: Intent Detection in the Age of LLMs

• Authors: Gaurav Arora, Shreya Jain, Srujana Merugu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01627
• Pdf URL: https://arxiv.org/pdf/2410.01627
• Copy Paste: [[2410.01627]] Intent Detection in the Age of LLMs(https://arxiv.org/abs/2410.01627)
• Keywords: transformer, generative, large language model
• Abstract: Intent detection is a critical component of task-oriented dialogue systems (TODS) which enables the identification of suitable actions to address user utterances at each dialog turn. Traditional approaches relied on computationally efficient supervised sentence transformer encoder models, which require substantial training data and struggle with out-of-scope (OOS) detection. The emergence of generative large language models (LLMs) with intrinsic world knowledge presents new opportunities to address these challenges. In this work, we adapt 7 SOTA LLMs using adaptive in-context learning and chain-of-thought prompting for intent detection, and compare their performance with contrastively fine-tuned sentence transformer (SetFit) models to highlight prediction quality and latency tradeoff. We propose a hybrid system using uncertainty based routing strategy to combine the two approaches that along with negative data augmentation results in achieving the best of both worlds ( i.e. within 2% of native LLM accuracy with 50% less latency). To better understand LLM OOS detection capabilities, we perform controlled experiments revealing that this capability is significantly influenced by the scope of intent labels and the size of the label space. We also introduce a two-step approach utilizing internal LLM representations, demonstrating empirical gains in OOS detection accuracy and F1-score by >5% for the Mistral-7B model.

Title: On The Adaptation of Unlimiformer for Decoder-Only Transformers

• Authors: Kian Ahrabian, Alon Benhaim, Barun Patra, Jay Pujara, Saksham Singhal, Xia Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01637
• Pdf URL: https://arxiv.org/pdf/2410.01637
• Copy Paste: [[2410.01637]] On The Adaptation of Unlimiformer for Decoder-Only Transformers(https://arxiv.org/abs/2410.01637)
• Keywords: transformer, large language model
• Abstract: One of the prominent issues stifling the current generation of large language models is their limited context length. Recent proprietary models such as GPT-4 and Claude 2 have introduced longer context lengths, 8k/32k and 100k, respectively; however, despite the efforts in the community, most common models, such as LLama-2, have a context length of 4k or less. Unlimiformer (Bertsch et al., 2023) is a recently popular vector-retrieval augmentation method that offloads cross-attention computations to a kNN index. However, its main limitation is incompatibility with decoder-only transformers out of the box. In this work, we explore practical considerations of adapting Unlimiformer to decoder-only transformers and introduce a series of modifications to overcome this limitation. Moreover, we expand the original experimental setup on summarization to include a new task (i.e., free-form Q&A) and an instruction-tuned model (i.e., a custom 6.7B GPT model). Our results showcase the effectiveness of these modifications on summarization, performing on par with a model with 2x the context length. Moreover, we discuss limitations and future directions for free-form Q&A and instruction-tuned models.

Title: Moral Alignment for LLM Agents

• Authors: Elizaveta Tennant, Stephen Hailes, Mirco Musolesi
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.01639
• Pdf URL: https://arxiv.org/pdf/2410.01639
• Copy Paste: [[2410.01639]] Moral Alignment for LLM Agents(https://arxiv.org/abs/2410.01639)
• Keywords: large language model
• Abstract: Decision-making agents based on pre-trained Large Language Models (LLMs) are increasingly being deployed across various domains of human activity. While their applications are currently rather specialized, several research efforts are under way to develop more generalist agents. As LLM-based systems become more agentic, their influence on human activity will grow and the transparency of this will decrease. Consequently, developing effective methods for aligning them to human values is vital. The prevailing practice in alignment often relies on human preference data (e.g., in RLHF or DPO), in which values are implicit and are essentially deduced from relative preferences over different model outputs. In this work, instead of relying on human feedback, we introduce the design of reward functions that explicitly encode core human values for Reinforcement Learning-based fine-tuning of foundation agent models. Specifically, we use intrinsic rewards for the moral alignment of LLM agents. We evaluate our approach using the traditional philosophical frameworks of Deontological Ethics and Utilitarianism, quantifying moral rewards for agents in terms of actions and consequences on the Iterated Prisoner's Dilemma (IPD) environment. We also show how moral fine-tuning can be deployed to enable an agent to unlearn a previously developed selfish strategy. Finally, we find that certain moral strategies learned on the IPD game generalize to several other matrix game environments. In summary, we demonstrate that fine-tuning with intrinsic rewards is a promising general solution for aligning LLM agents to human values, and it might represent a more transparent and cost-effective alternative to currently predominant alignment techniques.

Title: DeIDClinic: A Multi-Layered Framework for De-identification of Clinical Free-text Data

• Authors: Angel Paul, Dhivin Shaji, Lifeng Han, Warren Del-Pinto, Goran Nenadic
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01648
• Pdf URL: https://arxiv.org/pdf/2410.01648
• Copy Paste: [[2410.01648]] DeIDClinic: A Multi-Layered Framework for De-identification of Clinical Free-text Data(https://arxiv.org/abs/2410.01648)
• Keywords: privacy, protect
• Abstract: De-identification is important in protecting patients' privacy for healthcare text analytics. The MASK framework is one of the best on the de-identification shared task organised by n2c2/i2b2 challenges. This work enhances the MASK framework by integrating ClinicalBERT, a deep learning model specifically fine-tuned on clinical texts, alongside traditional de-identification methods like dictionary lookup and rule-based approaches. The system effectively identifies and either redacts or replaces sensitive identifiable entities within clinical documents, while also allowing users to customise the masked documents according to their specific needs. The integration of ClinicalBERT significantly improves the performance of entity recognition, achieving 0.9732 F1-score, especially for common entities such as names, dates, and locations. A risk assessment feature has also been developed, which analyses the uniqueness of context within documents to classify them into risk levels, guiding further de-identification efforts. While the system demonstrates strong overall performance, this work highlights areas for future improvement, including handling more complex entity occurrences and enhancing the system's adaptability to different clinical settings.

Title: shapiq: Shapley Interactions for Machine Learning

• Authors: Maximilian Muschalik, Hubert Baniecki, Fabian Fumagalli, Patrick Kolpaczki, Barbara Hammer, Eyke Hüllermeier
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01649
• Pdf URL: https://arxiv.org/pdf/2410.01649
• Copy Paste: [[2410.01649]] shapiq: Shapley Interactions for Machine Learning(https://arxiv.org/abs/2410.01649)
• Keywords: transformer
• Abstract: Originally rooted in game theory, the Shapley Value (SV) has recently become an important tool in machine learning research. Perhaps most notably, it is used for feature attribution and data valuation in explainable artificial intelligence. Shapley Interactions (SIs) naturally extend the SV and address its limitations by assigning joint contributions to groups of entities, which enhance understanding of black box machine learning models. Due to the exponential complexity of computing SVs and SIs, various methods have been proposed that exploit structural assumptions or yield probabilistic estimates given limited resources. In this work, we introduce shapiq, an open-source Python package that unifies state-of-the-art algorithms to efficiently compute SVs and any-order SIs in an application-agnostic framework. Moreover, it includes a benchmarking suite containing 11 machine learning applications of SIs with pre-computed games and ground-truth values to systematically assess computational performance across domains. For practitioners, shapiq is able to explain and visualize any-order feature interactions in predictions of models, including vision transformers, language models, as well as XGBoost and LightGBM with TreeSHAP-IQ. With shapiq, we extend shap beyond feature attributions and consolidate the application of SVs and SIs in machine learning that facilitates future research. The source code and documentation are available at this https URL.

Title: Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes

• Authors: Roussel Desmond Nzoyem, David A.W. Barton, Tom Deakin
• Subjects: cs.LG, math.DS
• Abstract URL: https://arxiv.org/abs/2410.01655
• Pdf URL: https://arxiv.org/pdf/2410.01655
• Copy Paste: [[2410.01655]] Extending Contextual Self-Modulation: Meta-Learning Across Modalities, Task Dimensionalities, and Data Regimes(https://arxiv.org/abs/2410.01655)
• Keywords: robust
• Abstract: Contextual Self-Modulation (CSM) is a potent regularization mechanism for the Neural Context Flow (NCF) framework which demonstrates powerful meta-learning of physical systems. However, CSM has limitations in its applicability across different modalities and in high-data regimes. In this work, we introduce two extensions: $i$CSM, which expands CSM to infinite-dimensional tasks, and StochasticNCF, which improves scalability. These extensions are demonstrated through comprehensive experimentation on a range of tasks, including dynamical systems with parameter variations, computer vision challenges, and curve fitting problems. $i$CSM embeds the contexts into an infinite-dimensional function space, as opposed to CSM which uses finite-dimensional context vectors. StochasticNCF enables the application of both CSM and $i$CSM to high-data scenarios by providing an unbiased approximation of meta-gradient updates through a sampled set of nearest environments. Additionally, we incorporate higher-order Taylor expansions via Taylor-Mode automatic differentiation, revealing that higher-order approximations do not necessarily enhance generalization. Finally, we demonstrate how CSM can be integrated into other meta-learning frameworks with FlashCAVIA, a computationally efficient extension of the CAVIA meta-learning framework (Zintgraf et al. 2019). FlashCAVIA outperforms its predecessor across various benchmarks and reinforces the utility of bi-level optimization techniques. Together, these contributions establish a robust framework for tackling an expanded spectrum of meta-learning tasks, offering practical insights for out-of-distribution generalization. Our open-sourced library, designed for flexible integration of self-modulation into contextual meta-learning workflows, is available at \url{this http URL}.

Title: Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering

• Authors: Klaus-Rudolf Kladny, Bernhard Schölkopf, Michael Muehlebach
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01660
• Pdf URL: https://arxiv.org/pdf/2410.01660
• Copy Paste: [[2410.01660]] Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering(https://arxiv.org/abs/2410.01660)
• Keywords: generative
• Abstract: Generative models lack rigorous statistical guarantees for their outputs and are therefore unreliable in safety-critical applications. In this work, we propose Sequential Conformal Prediction for Generative Models (SCOPE-Gen), a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee called conformal admissibility control. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example. To this end, our method first samples an initial set of i.i.d. examples from a black box generative model. Then, this set is iteratively pruned via so-called greedy filters. As a consequence of the iterative generation procedure, admissibility of the final prediction set factorizes as a Markov chain. This factorization is crucial, because it allows to control each factor separately, using conformal prediction. In comparison to prior work, our method demonstrates a large reduction in the number of admissibility evaluations during calibration. This reduction is important in safety-critical applications, where these evaluations must be conducted manually by domain experts and are therefore costly and time consuming. We highlight the advantages of our method in terms of admissibility evaluations and cardinality of the prediction sets through experiments in natural language generation and molecular graph extension tasks.

Title: Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding

• Authors: Yanming Liu, Xinyue Peng, Jiannan Cao, Shi Bo, Yanxin Shen, Xuhong Zhang, Sheng Cheng, Xun Wang, Jianwei Yin, Tianyu Du
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01671
• Pdf URL: https://arxiv.org/pdf/2410.01671
• Copy Paste: [[2410.01671]] Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding(https://arxiv.org/abs/2410.01671)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable capabilities in natural language processing; however, they still face difficulties when tasked with understanding lengthy contexts and executing effective question answering. These challenges often arise due to the complexity and ambiguity present in longer texts. To enhance the performance of LLMs in such scenarios, we introduce the Long Question Coreference Adaptation (LQCA) method. This innovative framework focuses on coreference resolution tailored to long contexts, allowing the model to identify and manage references effectively. The LQCA method encompasses four key steps: resolving coreferences within sub-documents, computing the distances between mentions, defining a representative mention for coreference, and answering questions through mention replacement. By processing information systematically, the framework provides easier-to-handle partitions for LLMs, promoting better understanding. Experimental evaluations on a range of LLMs and datasets have yielded positive results, with a notable improvements on OpenAI-o1-mini and GPT-4o models, highlighting the effectiveness of leveraging coreference resolution to bridge context gaps in question answering.

Title: Trying to be human: Linguistic traces of stochastic empathy in language models

• Authors: Bennett Kleinberg, Jari Zegers, Jonas Festor, Stefana Vida, Julian Präsent, Riccardo Loconte, Sanne Peereboom
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01675
• Pdf URL: https://arxiv.org/pdf/2410.01675
• Copy Paste: [[2410.01675]] Trying to be human: Linguistic traces of stochastic empathy in language models(https://arxiv.org/abs/2410.01675)
• Keywords: large language model
• Abstract: Differentiating between generated and human-written content is important for navigating the modern world. Large language models (LLMs) are crucial drivers behind the increased quality of computer-generated content. Reportedly, humans find it increasingly difficult to identify whether an AI model generated a piece of text. Our work tests how two important factors contribute to the human vs AI race: empathy and an incentive to appear human. We address both aspects in two experiments: human participants and a state-of-the-art LLM wrote relationship advice (Study 1, n=530) or mere descriptions (Study 2, n=610), either instructed to be as human as possible or not. New samples of humans (n=428 and n=408) then judged the texts' source. Our findings show that when empathy is required, humans excel. Contrary to expectations, instructions to appear human were only effective for the LLM, so the human advantage diminished. Computational text analysis revealed that LLMs become more human because they may have an implicit representation of what makes a text human and effortlessly apply these heuristics. The model resorts to a conversational, self-referential, informal tone with a simpler vocabulary to mimic stochastic empathy. We discuss these findings in light of recent claims on the on-par performance of LLMs.

Title: Open3DTrack: Towards Open-Vocabulary 3D Multi-Object Tracking

• Authors: Ayesha Ishaq, Mohamed El Amine Boudjoghra, Jean Lahoud, Fahad Shahbaz Khan, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.01678
• Pdf URL: https://arxiv.org/pdf/2410.01678
• Copy Paste: [[2410.01678]] Open3DTrack: Towards Open-Vocabulary 3D Multi-Object Tracking(https://arxiv.org/abs/2410.01678)
• Keywords: robust
• Abstract: 3D multi-object tracking plays a critical role in autonomous driving by enabling the real-time monitoring and prediction of multiple objects' movements. Traditional 3D tracking systems are typically constrained by predefined object categories, limiting their adaptability to novel, unseen objects in dynamic environments. To address this limitation, we introduce open-vocabulary 3D tracking, which extends the scope of 3D tracking to include objects beyond predefined categories. We formulate the problem of open-vocabulary 3D tracking and introduce dataset splits designed to represent various open-vocabulary scenarios. We propose a novel approach that integrates open-vocabulary capabilities into a 3D tracking framework, allowing for generalization to unseen object classes. Our method effectively reduces the performance gap between tracking known and novel objects through strategic adaptation. Experimental results demonstrate the robustness and adaptability of our method in diverse outdoor driving scenarios. To the best of our knowledge, this work is the first to address open-vocabulary 3D tracking, presenting a significant advancement for autonomous systems in real-world settings. Code, trained models, and dataset splits are available publicly.

Title: VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment

• Authors: Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, Nicolas Le Roux
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01679
• Pdf URL: https://arxiv.org/pdf/2410.01679
• Copy Paste: [[2410.01679]] VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment(https://arxiv.org/abs/2410.01679)
• Keywords: large language model
• Abstract: Large language models (LLMs) are increasingly applied to complex reasoning tasks that require executing several complex steps before receiving any reward. Properly assigning credit to these steps is essential for enhancing model performance. Proximal Policy Optimization (PPO), a state-of-the-art reinforcement learning (RL) algorithm used for LLM finetuning, employs value networks to tackle credit assignment. However, value networks face challenges in predicting the expected cumulative rewards accurately in complex reasoning tasks, often leading to high-variance updates and suboptimal performance. In this work, we systematically evaluate the efficacy of value networks and reveal their significant shortcomings in reasoning-heavy LLM tasks, showing that they barely outperform a random baseline when comparing alternative steps. To address this, we propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates, bypassing the need for large value networks. Our method consistently outperforms PPO and other RL-free baselines across MATH and GSM8K datasets with fewer gradient updates (up to 9x), less wall-clock time (up to 3.0x). These results emphasize the importance of accurate credit assignment in RL finetuning of LLM and demonstrate VinePPO's potential as a superior alternative.

Title: Positional Attention: Out-of-Distribution Generalization and Expressivity for Neural Algorithmic Reasoning

• Authors: Artur Back de Luca, George Giapitzakis, Shenghao Yang, Petar Veličković, Kimon Fountoulakis
• Subjects: cs.LG, cs.AI, cs.DS
• Abstract URL: https://arxiv.org/abs/2410.01686
• Pdf URL: https://arxiv.org/pdf/2410.01686
• Copy Paste: [[2410.01686]] Positional Attention: Out-of-Distribution Generalization and Expressivity for Neural Algorithmic Reasoning(https://arxiv.org/abs/2410.01686)
• Keywords: transformer
• Abstract: There has been a growing interest in the ability of neural networks to solve algorithmic tasks, such as arithmetic, summary statistics, and sorting. While state-of-the-art models like Transformers have demonstrated good generalization performance on in-distribution tasks, their out-of-distribution (OOD) performance is poor when trained end-to-end. In this paper, we focus on value generalization, a common instance of OOD generalization where the test distribution has the same input sequence length as the training distribution, but the value ranges in the training and test distributions do not necessarily overlap. To address this issue, we propose that using fixed positional encodings to determine attention weights-referred to as positional attention-enhances empirical OOD performance while maintaining expressivity. We support our claim about expressivity by proving that Transformers with positional attention can effectively simulate parallel algorithms.

Title: FactAlign: Long-form Factuality Alignment of Large Language Models

• Authors: Chao-Wei Huang, Yun-Nung Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01691
• Pdf URL: https://arxiv.org/pdf/2410.01691
• Copy Paste: [[2410.01691]] FactAlign: Long-form Factuality Alignment of Large Language Models(https://arxiv.org/abs/2410.01691)
• Keywords: large language model
• Abstract: Large language models have demonstrated significant potential as the next-generation information access engines. However, their reliability is hindered by issues of hallucination and generating non-factual content. This is particularly problematic in long-form responses, where assessing and ensuring factual accuracy is complex. In this paper, we address this gap by proposing FactAlign, a novel alignment framework designed to enhance the factuality of LLMs' long-form responses while maintaining their helpfulness. We introduce fKTO, a fine-grained, sentence-level alignment algorithm that extends the Kahneman-Tversky Optimization (KTO) alignment method. Leveraging recent advances in automatic factuality evaluation, FactAlign utilizes fine-grained factuality assessments to guide the alignment process. Our experiments on open-domain prompts and information-seeking questions demonstrate that FactAlign significantly improves the factual accuracy of LLM responses while also improving their helpfulness. Further analyses identify that FactAlign is capable of training LLMs to provide more information without losing factual precision, thus improving the factual F1 score. Our source code, datasets, and trained models are publicly available at this https URL

Title: MOREL: Enhancing Adversarial Robustness through Multi-Objective Representation Learning

• Authors: Sedjro Salomon Hotegni, Sebastian Peitz
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01697
• Pdf URL: https://arxiv.org/pdf/2410.01697
• Copy Paste: [[2410.01697]] MOREL: Enhancing Adversarial Robustness through Multi-Objective Representation Learning(https://arxiv.org/abs/2410.01697)
• Keywords: defense, attack, robust
• Abstract: Extensive research has shown that deep neural networks (DNNs) are vulnerable to slight adversarial perturbations$-$small changes to the input data that appear insignificant but cause the model to produce drastically different outputs. In addition to augmenting training data with adversarial examples generated from a specific attack method, most of the current defense strategies necessitate modifying the original model architecture components to improve robustness or performing test-time data purification to handle adversarial attacks. In this work, we demonstrate that strong feature representation learning during training can significantly enhance the original model's robustness. We propose MOREL, a multi-objective feature representation learning approach, encouraging classification models to produce similar features for inputs within the same class, despite perturbations. Our training method involves an embedding space where cosine similarity loss and multi-positive contrastive loss are used to align natural and adversarial features from the model encoder and ensure tight clustering. Concurrently, the classifier is motivated to achieve accurate predictions. Through extensive experiments, we demonstrate that our approach significantly enhances the robustness of DNNs against white-box and black-box adversarial attacks, outperforming other methods that similarly require no architectural changes or test-time data purification. Our code is available at this https URL

Title: An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings

• Authors: Soham Govande
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01704
• Pdf URL: https://arxiv.org/pdf/2410.01704
• Copy Paste: [[2410.01704]] An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings(https://arxiv.org/abs/2410.01704)
• Keywords: extraction
• Abstract: There is a growing need for pluralistic alignment methods that can steer language models towards individual attributes and preferences. One such method, Self-Supervised Alignment with Mutual Information (SAMI), uses conditional mutual information to encourage the connection between behavioral preferences and model responses. We conduct two experiments exploring SAMI in multi-task settings. First, we compare SAMI to Direct Preference Optimization (DPO) on a multi-task benchmark (MT-Bench), using a stronger model to generate training data for a weaker one across diverse categories (humanities, STEM, extraction, coding, math, reasoning, and roleplay). Our results indicate that one iteration of SAMI has a 57% win rate against DPO, with significant variation in performance between task categories. Second, we examine SAMI's impact on mathematical accuracy (GSM-8K) relative to supervised fine-tuning (SFT). While SAMI increases zero-shot performance by 1.1%, SFT is more effective with a 3.2% boost. However, SAMI shows interesting scaling trends. When given 10 attempts, SAMI improves accuracy by 3.9%, while SFT achieves a 10.1% increase. Combining SAMI with SFT yields an additional improvement of 1.3% in multi-attempt settings, though single-attempt accuracy remains unchanged.

Title: Interpretable Contrastive Monte Carlo Tree Search Reasoning

• Authors: Zitian Gao, Boye Niu, Xuzheng He, Haotian Xu, Hongzhang Liu, Aiwei Liu, Xuming Hu, Lijie Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01707
• Pdf URL: https://arxiv.org/pdf/2410.01707
• Copy Paste: [[2410.01707]] Interpretable Contrastive Monte Carlo Tree Search Reasoning(https://arxiv.org/abs/2410.01707)
• Keywords: interpretability, large language model
• Abstract: We propose SC-MCTS*: a novel Monte Carlo Tree Search (MCTS) reasoning algorithm for Large Language Models (LLMs), significantly improves both reasoning accuracy and speed. Our motivation comes from: 1. Previous MCTS LLM reasoning works often overlooked its biggest drawback--slower speed compared to CoT; 2. Previous research mainly used MCTS as a tool for LLM reasoning on various tasks with limited quantitative analysis or ablation studies of its components from reasoning interpretability perspective. 3. The reward model is the most crucial component in MCTS, however previous work has rarely conducted in-depth study or improvement of MCTS's reward models. Thus, we conducted extensive ablation studies and quantitative analysis on components of MCTS, revealing the impact of each component on the MCTS reasoning performance of LLMs. Building on this, (i) we designed a highly interpretable reward model based on the principle of contrastive decoding and (ii) achieved an average speed improvement of 51.9% per node using speculative decoding. Additionally, (iii) we improved UCT node selection strategy and backpropagation used in previous works, resulting in significant performance improvement. We outperformed o1-mini by an average of 17.4% on the Blocksworld multi-step reasoning dataset using Llama-3.1-70B with SC-MCTS*.

Title: Examining the Role of Relationship Alignment in Large Language Models

• Authors: Kristen M. Altenburger, Hongda Jiang, Robert E. Kraut, Yi-Chia Wang, Jane Dwivedi-Yu
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2410.01708
• Pdf URL: https://arxiv.org/pdf/2410.01708
• Copy Paste: [[2410.01708]] Examining the Role of Relationship Alignment in Large Language Models(https://arxiv.org/abs/2410.01708)
• Keywords: generative, large language model
• Abstract: The rapid development and deployment of Generative AI in social settings raise important questions about how to optimally personalize them for users while maintaining accuracy and realism. Based on a Facebook public post-comment dataset, this study evaluates the ability of Llama 3.0 (70B) to predict the semantic tones across different combinations of a commenter's and poster's gender, age, and friendship closeness and to replicate these differences in LLM-generated comments. The study consists of two parts: Part I assesses differences in semantic tones across social relationship categories, and Part II examines the similarity between comments generated by Llama 3.0 (70B) and human comments from Part I given public Facebook posts as input. Part I results show that including social relationship information improves the ability of a model to predict the semantic tone of human comments. However, Part II results show that even without including social context information in the prompt, LLM-generated comments and human comments are equally sensitive to social context, suggesting that LLMs can comprehend semantics from the original post alone. When we include all social relationship information in the prompt, the similarity between human comments and LLM-generated comments decreases. This inconsistency may occur because LLMs did not include social context information as part of their training data. Together these results demonstrate the ability of LLMs to comprehend semantics from the original post and respond similarly to human comments, but also highlights their limitations in generalizing personalized comments through prompting alone.

Title: Meta-TTT: A Meta-learning Minimax Framework For Test-Time Training

• Authors: Chen Tao, Li Shen, Soumik Mondal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01709
• Pdf URL: https://arxiv.org/pdf/2410.01709
• Copy Paste: [[2410.01709]] Meta-TTT: A Meta-learning Minimax Framework For Test-Time Training(https://arxiv.org/abs/2410.01709)
• Keywords: robust
• Abstract: Test-time domain adaptation is a challenging task that aims to adapt a pre-trained model to limited, unlabeled target data during inference. Current methods that rely on self-supervision and entropy minimization underperform when the self-supervised learning (SSL) task does not align well with the primary objective. Additionally, minimizing entropy can lead to suboptimal solutions when there is limited diversity within minibatches. This paper introduces a meta-learning minimax framework for test-time training on batch normalization (BN) layers, ensuring that the SSL task aligns with the primary task while addressing minibatch overfitting. We adopt a mixed-BN approach that interpolates current test batch statistics with the statistics from source domains and propose a stochastic domain synthesizing method to improve model generalization and robustness to domain shifts. Extensive experiments demonstrate that our method surpasses state-of-the-art techniques across various domain adaptation and generalization benchmarks, significantly enhancing the pre-trained model's robustness on unseen domains.

Title: COMUNI: Decomposing Common and Unique Video Signals for Diffusion-based Video Generation

• Authors: Mingzhen Sun, Weining Wang, Xinxin Zhu, Jing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01718
• Pdf URL: https://arxiv.org/pdf/2410.01718
• Copy Paste: [[2410.01718]] COMUNI: Decomposing Common and Unique Video Signals for Diffusion-based Video Generation(https://arxiv.org/abs/2410.01718)
• Keywords: diffusion, generative
• Abstract: Since videos record objects moving coherently, adjacent video frames have commonness (similar object appearances) and uniqueness (slightly changed postures). To prevent redundant modeling of common video signals, we propose a novel diffusion-based framework, named COMUNI, which decomposes the COMmon and UNIque video signals to enable efficient video generation. Our approach separates the decomposition of video signals from the task of video generation, thus reducing the computation complexity of generative models. In particular, we introduce CU-VAE to decompose video signals and encode them into latent features. To train CU-VAE in a self-supervised manner, we employ a cascading merge module to reconstitute video signals and a time-agnostic video decoder to reconstruct video frames. Then we propose CU-LDM to model latent features for video generation, which adopts two specific diffusion streams to simultaneously model the common and unique latent features. We further utilize additional joint modules for cross modeling of the common and unique latent features, and a novel position embedding method to ensure the content consistency and motion coherence of generated videos. The position embedding method incorporates spatial and temporal absolute position information into the joint modules. Extensive experiments demonstrate the necessity of decomposing common and unique video signals for video generation and the effectiveness and efficiency of our proposed method.

Title: HarmoniCa: Harmonizing Training and Inference for Better Feature Cache in Diffusion Transformer Acceleration

• Authors: Yushi Huang, Zining Wang, Ruihao Gong, Jing Liu, Xinjie Zhang, Jinyang Guo, Xianglong Liu, Jun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01723
• Pdf URL: https://arxiv.org/pdf/2410.01723
• Copy Paste: [[2410.01723]] HarmoniCa: Harmonizing Training and Inference for Better Feature Cache in Diffusion Transformer Acceleration(https://arxiv.org/abs/2410.01723)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion Transformers (DiTs) have gained prominence for outstanding scalability and extraordinary performance in generative tasks. However, their considerable inference costs impede practical deployment. The feature cache mechanism, which involves storing and retrieving redundant computations across timesteps, holds promise for reducing per-step inference time in diffusion models. Most existing caching methods for DiT are manually designed. Although the learning-based approach attempts to optimize strategies adaptively, it suffers from discrepancies between training and inference, which hampers both the performance and acceleration ratio. Upon detailed analysis, we pinpoint that these discrepancies primarily stem from two aspects: (1) Prior Timestep Disregard, where training ignores the effect of cache usage at earlier timesteps, and (2) Objective Mismatch, where the training target (align predicted noise in each timestep) deviates from the goal of inference (generate the high-quality image). To alleviate these discrepancies, we propose HarmoniCa, a novel method that Harmonizes training and inference with a novel learning-based Caching framework built upon Step-Wise Denoising Training (SDT) and Image Error Proxy-Guided Objective (IEPO). Compared to the traditional training paradigm, the newly proposed SDT maintains the continuity of the denoising process, enabling the model to leverage information from prior timesteps during training, similar to the way it operates during inference. Furthermore, we design IEPO, which integrates an efficient proxy mechanism to approximate the final image error caused by reusing the cached feature. Therefore, IEPO helps balance final image quality and cache utilization, resolving the issue of training that only considers the impact of cache usage on the predicted output at each timestep.

Title: Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting

• Authors: Longyu Feng, Mengze Hong, Chen Jason Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01724
• Pdf URL: https://arxiv.org/pdf/2410.01724
• Copy Paste: [[2410.01724]] Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting(https://arxiv.org/abs/2410.01724)
• Keywords: robust, large language model
• Abstract: Batch prompting is a common technique in large language models (LLMs) used to process multiple inputs simultaneously, aiming to improve computational efficiency. However, as batch sizes increase, performance degradation often occurs due to the model's difficulty in handling lengthy context inputs. Existing methods that attempt to mitigate these issues rely solely on batch data arrangement and majority voting rather than improving the design of the batch prompt itself. In this paper, we address these limitations by proposing "Auto-Demo Prompting," a novel approach that leverages the question-output pairs from earlier questions within a batch as demonstrations for subsequent answer inference. We provide a formal theoretical analysis of how Auto-Demo Prompting functions within the autoregressive generation process of LLMs, illustrating how it utilizes prior outputs to optimize the model's internal representations. Our method effectively bridges the gap between batch prompting and few-shot prompting, enhancing performance with only a slight compromise in token usage. Experimental results across five NLP tasks demonstrate its effectiveness in mitigating performance degradation and occasionally outperforming single prompts. Furthermore, it opens new avenues for applying few-shot learning techniques, such as demonstration selection, within batch prompting, making it a robust solution for real-world applications.

Title: Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing

• Authors: Yilmazcan Ozyurt, Stefan Feuerriegel, Mrinmaya Sachan
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01727
• Pdf URL: https://arxiv.org/pdf/2410.01727
• Copy Paste: [[2410.01727]] Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing(https://arxiv.org/abs/2410.01727)
• Keywords: large language model
• Abstract: Knowledge tracing (KT) is a popular approach for modeling students' learning progress over time, which can enable more personalized and adaptive learning. However, existing KT approaches face two major limitations: (1) they rely heavily on expert-defined knowledge concepts (KCs) in questions, which is time-consuming and prone to errors; and (2) KT methods tend to overlook the semantics of both questions and the given KCs. In this work, we address these challenges and present KCQRL, a framework for automated knowledge concept annotation and question representation learning that can improve the effectiveness of any existing KT model. First, we propose an automated KC annotation process using large language models (LLMs), which generates question solutions and then annotates KCs in each solution step of the questions. Second, we introduce a contrastive learning approach to generate semantically rich embeddings for questions and solution steps, aligning them with their associated KCs via a tailored false negative elimination approach. These embeddings can be readily integrated into existing KT models, replacing their randomly initialized embeddings. We demonstrate the effectiveness of KCQRL across 15 KT algorithms on two large real-world Math learning datasets, where we achieve consistent performance improvements.

Title: Evaluating Robustness of Reward Models for Mathematical Reasoning

• Authors: Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, Jinyoung Yeo
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01729
• Pdf URL: https://arxiv.org/pdf/2410.01729
• Copy Paste: [[2410.01729]] Evaluating Robustness of Reward Models for Mathematical Reasoning(https://arxiv.org/abs/2410.01729)
• Keywords: robust
• Abstract: Reward models are key in reinforcement learning from human feedback (RLHF) systems, aligning the model behavior with human preferences. Particularly in the math domain, there have been plenty of studies using reward models to align policies for improving reasoning capabilities. Recently, as the importance of reward models has been emphasized, RewardBench is proposed to understand their behavior. However, we figure out that the math subset of RewardBench has different representations between chosen and rejected completions, and relies on a single comparison, which may lead to unreliable results as it only see an isolated case. Therefore, it fails to accurately present the robustness of reward models, leading to a misunderstanding of its performance and potentially resulting in reward hacking. In this work, we introduce a new design for reliable evaluation of reward models, and to validate this, we construct RewardMATH, a benchmark that effectively represents the robustness of reward models in mathematical reasoning tasks. We demonstrate that the scores on RewardMATH strongly correlate with the results of optimized policy and effectively estimate reward overoptimization, whereas the existing benchmark shows almost no correlation. The results underscore the potential of our design to enhance the reliability of evaluation, and represent the robustness of reward model. We make our code and data publicly available.

Title: Visual Perception in Text Strings

• Authors: Qi Jia, Xiang Yue, Shanshan Huang, Ziheng Qin, Yizhu Liu, Bill Yuchen Lin, Yang You
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01733
• Pdf URL: https://arxiv.org/pdf/2410.01733
• Copy Paste: [[2410.01733]] Visual Perception in Text Strings(https://arxiv.org/abs/2410.01733)
• Keywords: large language model
• Abstract: Understanding visual semantics embedded in consecutive characters is a crucial capability for both large language models (LLMs) and multi-modal large language models (MLLMs). This type of artifact possesses the unique characteristic that identical information can be readily formulated in both texts and images, making them a significant proxy for analyzing modern LLMs' and MLLMs' capabilities in modality-agnostic vision understanding. In this work, we select ASCII art as a representative artifact, where the lines and brightness used to depict each concept are rendered by characters, and we frame the problem as an ASCII art recognition task. We benchmark model performance on this task by constructing an evaluation dataset with an elaborate categorization tree and also collect a training set to elicit the models' visual perception ability. Through a comprehensive analysis of dozens of models, results reveal that although humans can achieve nearly 100% accuracy, the state-of-the-art LLMs and MLLMs lag far behind. Models are capable of recognizing concepts depicted in the ASCII arts given only text inputs indicated by over 60% accuracy for some concepts, but most of them achieves merely around 30% accuracy when averaged across all categories. When provided with images as inputs, GPT-4o gets 82.68%, outperforming the strongest open-source MLLM by 21.95%. Although models favor different kinds of ASCII art depending on the modality provided, none of the MLLMs successfully benefit when both modalities are supplied simultaneously. Moreover, supervised fine-tuning helps improve models' accuracy especially when provided with the image modality, but also highlights the need for better training techniques to enhance the information fusion among modalities.

Title: LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits

• Authors: Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01735
• Pdf URL: https://arxiv.org/pdf/2410.01735
• Copy Paste: [[2410.01735]] LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits(https://arxiv.org/abs/2410.01735)
• Keywords: robust
• Abstract: Reward Models (RMs) play a crucial role in aligning LLMs with human preferences, enhancing their performance by ranking outputs during inference or iterative training. However, the degree to which an RM generalizes to new tasks is often not known a priori (e.g. some RMs may excel at scoring creative writing vs. math reasoning). Therefore, using only one fixed RM while training LLMs can be suboptimal. Moreover, optimizing LLMs with multiple RMs simultaneously can be prohibitively computationally-intensive and challenging due to conflicting signals from different RMs, potentially degrading performance. To address these challenges, we introduce LASeR (Learning to Adaptively Select Rewards), which iteratively trains LLMs using multiple RMs, selecting and utilizing the most well-suited RM for each instance to rank outputs and generate preference data, framed as a multi-armed bandit problem. Our results on commonsense and math reasoning tasks demonstrate that LASeR can boost iterative LLM optimization by optimizing for multiple RMs, improving the absolute average accuracy of Llama-3-8B over three datasets by 2.67% over training with ensemble RM scores while also showing superior training efficiency (e.g., a 2x speedup). Moreover, on WildChat, a benchmark of instruction-following prompts, we find that using Llama-3-8B LASeR leads to a 71.45% AlpacaEval win rate over sequentially optimizing multiple RMs. Extending to long-context generation tasks, we find that on Llama-3-8B, LASeR achieves an average improvement of 2.64 F1 and 2.42 F1 on single- and multi-document QA over random RM selection when used with best-of-n sampling. LASeR is robust to noisy rewards and generalizes to multiple settings. Finally, LASeR's RM selection changes depending on the underlying task or instance and we verify the presence of conflicting preferences from multiple RMs that can be mitigated using LASeR.

Title: RADAR: Robust Two-stage Modality-incomplete Industrial Anomaly Detection

• Authors: Bingchen Miao, Wenqiao Zhang, Juncheng Li, Siliang Tang, Zhaocheng Li, Haochen Shi, Jun Xiao, Yueting Zhuang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.01737
• Pdf URL: https://arxiv.org/pdf/2410.01737
• Copy Paste: [[2410.01737]] RADAR: Robust Two-stage Modality-incomplete Industrial Anomaly Detection(https://arxiv.org/abs/2410.01737)
• Keywords: robust, transformer
• Abstract: Multimodal Industrial Anomaly Detection (MIAD), utilizing 3D point clouds and 2D RGB images to identify the abnormal region of products, plays a crucial role in industrial quality inspection. However, the conventional MIAD setting presupposes that all 2D and 3D modalities are paired, overlooking the fact that multimodal data collected from the real world is often imperfect due to missing modalities. Consequently, MIAD models that demonstrate robustness against modal-incomplete data are highly desirable in practice. To address this practical challenge, we introduce a first-of-its-kind study that comprehensively investigates Modality-Incomplete Industrial Anomaly Detection (MIIAD), to consider the imperfect learning environment in which the multimodal information may be incomplete. Not surprisingly, we discovered that most existing MIAD approaches are inadequate for addressing MIIAD challenges, leading to significant performance degradation on the MIIAD benchmark we developed. In this paper, we propose a novel two-stage Robust modAlity-imcomplete fusing and Detecting frAmewoRk, abbreviated as RADAR. Our bootstrapping philosophy is to enhance two stages in MIIAD, improving the robustness of the Multimodal Transformer: i) In feature fusion, we first explore learning modality-incomplete instruction, guiding the pre-trained Multimodal Transformer to robustly adapt to various modality-incomplete scenarios, and implement adaptive parameter learning based on a HyperNetwork; ii) In anomaly detection, we construct a real-pseudo hybrid module to highlight the distinctiveness of modality combinations, further enhancing the robustness of the MIIAD model. Our experimental results demonstrate that the proposed RADAR significantly surpasses conventional MIAD methods in terms of effectiveness and robustness on our newly created MIIAD dataset, underscoring its practical application value.

Title: VitaGlyph: Vitalizing Artistic Typography with Flexible Dual-branch Diffusion Models

• Authors: Kailai Feng, Yabo Zhang, Haodong Yu, Zhilong Ji, Jinfeng Bai, Hongzhi Zhang, Wangmeng Zuo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01738
• Pdf URL: https://arxiv.org/pdf/2410.01738
• Copy Paste: [[2410.01738]] VitaGlyph: Vitalizing Artistic Typography with Flexible Dual-branch Diffusion Models(https://arxiv.org/abs/2410.01738)
• Keywords: diffusion, large language model
• Abstract: Artistic typography is a technique to visualize the meaning of input character in an imaginable and readable manner. With powerful text-to-image diffusion models, existing methods directly design the overall geometry and texture of input character, making it challenging to ensure both creativity and legibility. In this paper, we introduce a dual-branch and training-free method, namely VitaGlyph, enabling flexible artistic typography along with controllable geometry change to maintain the readability. The key insight of VitaGlyph is to treat input character as a scene composed of Subject and Surrounding, followed by rendering them under varying degrees of geometry transformation. The subject flexibly expresses the essential concept of input character, while the surrounding enriches relevant background without altering the shape. Specifically, we implement VitaGlyph through a three-phase framework: (i) Knowledge Acquisition leverages large language models to design text descriptions of subject and surrounding. (ii) Regional decomposition detects the part that most matches the subject description and divides input glyph image into subject and surrounding regions. (iii) Typography Stylization firstly refines the structure of subject region via Semantic Typography, and then separately renders the textures of Subject and Surrounding regions through Controllable Compositional Generation. Experimental results demonstrate that VitaGlyph not only achieves better artistry and readability, but also manages to depict multiple customize concepts, facilitating more creative and pleasing artistic typography generation. Our code will be made publicly at this https URL.

Title: LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks

• Authors: Mengzhao Jia, Wenhao Yu, Kaixin Ma, Tianqing Fang, Zhihan Zhang, Siru Ouyang, Hongming Zhang, Meng Jiang, Dong Yu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01744
• Pdf URL: https://arxiv.org/pdf/2410.01744
• Copy Paste: [[2410.01744]] LEOPARD : A Vision Language Model For Text-Rich Multi-Image Tasks(https://arxiv.org/abs/2410.01744)
• Keywords: large language model
• Abstract: Text-rich images, where text serves as the central visual element guiding the overall understanding, are prevalent in real-world applications, such as presentation slides, scanned documents, and webpage snapshots. Tasks involving multiple text-rich images are especially challenging, as they require not only understanding the content of individual images but reasoning about inter-relationships and logical flows across multiple visual inputs. Despite the importance of these scenarios, current multimodal large language models (MLLMs) struggle to handle such tasks due to two key challenges: (1) the scarcity of high-quality instruction tuning datasets for text-rich multi-image scenarios, and (2) the difficulty in balancing image resolution with visual feature sequence length. To address these challenges, we propose \OurMethod, a MLLM designed specifically for handling vision-language tasks involving multiple text-rich images. First, we curated about one million high-quality multimodal instruction-tuning data, tailored to text-rich, multi-image scenarios. Second, we developed an adaptive high-resolution multi-image encoding module to dynamically optimize the allocation of visual sequence length based on the original aspect ratios and resolutions of the input images. Experiments across a wide range of benchmarks demonstrate our model's superior capabilities in text-rich, multi-image evaluations and competitive performance in general domain evaluations.

Title: PreND: Enhancing Intrinsic Motivation in Reinforcement Learning through Pre-trained Network Distillation

• Authors: Mohammadamin Davoodabadi, Negin Hashemi Dijujin, Mahdieh Soleymani Baghshah
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01745
• Pdf URL: https://arxiv.org/pdf/2410.01745
• Copy Paste: [[2410.01745]] PreND: Enhancing Intrinsic Motivation in Reinforcement Learning through Pre-trained Network Distillation(https://arxiv.org/abs/2410.01745)
• Keywords: robust
• Abstract: Intrinsic motivation, inspired by the psychology of developmental learning in infants, stimulates exploration in agents without relying solely on sparse external rewards. Existing methods in reinforcement learning like Random Network Distillation (RND) face significant limitations, including (1) relying on raw visual inputs, leading to a lack of meaningful representations, (2) the inability to build a robust latent space, (3) poor target network initialization and (4) rapid degradation of intrinsic rewards. In this paper, we introduce Pre-trained Network Distillation (PreND), a novel approach to enhance intrinsic motivation in reinforcement learning (RL) by improving upon the widely used prediction-based method, RND. PreND addresses these challenges by incorporating pre-trained representation models into both the target and predictor networks, resulting in more meaningful and stable intrinsic rewards, while enhancing the representation learned by the model. We also tried simple but effective variants of the predictor network optimization by controlling the learning rate. Through experiments on the Atari domain, we demonstrate that PreND significantly outperforms RND, offering a more robust intrinsic motivation signal that leads to better exploration, improving overall performance and sample efficiency. This research highlights the importance of target and predictor networks representation in prediction-based intrinsic motivation, setting a new direction for improving RL agents' learning efficiency in sparse reward environments.

Title: AssessITS: Integrating procedural guidelines and practical evaluation metrics for organizational IT and Cybersecurity risk assessment

• Authors: Mir Mehedi Rahman, Naresh Kshetri, Sayed Abu Sayeed, Md Masud Rana
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.01750
• Pdf URL: https://arxiv.org/pdf/2410.01750
• Copy Paste: [[2410.01750]] AssessITS: Integrating procedural guidelines and practical evaluation metrics for organizational IT and Cybersecurity risk assessment(https://arxiv.org/abs/2410.01750)
• Keywords: security, robust
• Abstract: In today's digitally driven landscape, robust Information Technology (IT) risk assessment practices are essential for safeguarding systems, digital communication, and data. This paper introduces 'AssessITS', an actionable method designed to provide organizations with comprehensive guidelines for conducting IT and cybersecurity risk assessments. Drawing extensively from NIST 800-30 Rev 1, COBIT 5, and ISO 31000, 'AssessITS' bridges the gap between high-level theoretical standards and practical implementation challenges. The paper outlines a step-by-step methodology that organizations can simply adopt to systematically identify, analyze, and mitigate IT risks. By simplifying complex principles into actionable procedures, this framework equips practitioners with the tools needed to perform risk assessments independently, without too much reliance on external vendors. The guidelines are developed to be straightforward, integrating practical evaluation metrics that allow for the precise quantification of asset values, threat levels, vulnerabilities, and impacts on confidentiality, integrity, and availability. This approach ensures that the risk assessment process is not only comprehensive but also accessible, enabling decision-makers to implement effective risk mitigation strategies customized to their unique operational contexts. 'AssessITS' aims to enable organizations to enhance their IT security strength through practical, actionable guidance based on internationally recognized standards.

Title: ImageFolder: Autoregressive Image Generation with Folded Tokens

• Authors: Xiang Li, Hao Chen, Kai Qiu, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, Zhe Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01756
• Pdf URL: https://arxiv.org/pdf/2410.01756
• Copy Paste: [[2410.01756]] ImageFolder: Autoregressive Image Generation with Folded Tokens(https://arxiv.org/abs/2410.01756)
• Keywords: diffusion, generative
• Abstract: Image tokenizers are crucial for visual generative models, e.g., diffusion models (DMs) and autoregressive (AR) models, as they construct the latent representation for modeling. Increasing token length is a common approach to improve the image reconstruction quality. However, tokenizers with longer token lengths are not guaranteed to achieve better generation quality. There exists a trade-off between reconstruction and generation quality regarding token length. In this paper, we investigate the impact of token length on both image reconstruction and generation and provide a flexible solution to the tradeoff. We propose ImageFolder, a semantic tokenizer that provides spatially aligned image tokens that can be folded during autoregressive modeling to improve both generation efficiency and quality. To enhance the representative capability without increasing token length, we leverage dual-branch product quantization to capture different contexts of images. Specifically, semantic regularization is introduced in one branch to encourage compacted semantic information while another branch is designed to capture the remaining pixel-level details. Extensive experiments demonstrate the superior quality of image generation and shorter token length with ImageFolder tokenizer.

Title: LightSC: The Making of a Usable Security Classification Tool for DevSecOps

• Authors: Manish Shrestha, Christian Johansen, Johanna Johansen
• Subjects: cs.CR, cs.HC, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.01762
• Pdf URL: https://arxiv.org/pdf/2410.01762
• Copy Paste: [[2410.01762]] LightSC: The Making of a Usable Security Classification Tool for DevSecOps(https://arxiv.org/abs/2410.01762)
• Keywords: security
• Abstract: DevSecOps, as the extension of DevOps with security training and tools, has become a popular way of developing modern software, especially in the Internet of Things arena, due to its focus on rapid development, with short release cycles, involving the user/client very closely. Security classification methods, on the other hand, are heavy and slow processes that require high expertise in security, the same as in other similar areas such as risk analysis or certification. As such, security classification methods are hardly compatible with the DevSecOps culture, which to the contrary, has moved away from the traditional style of penetration testing done only when the software product is in the final stages or already deployed. In this work, we first propose five principles for a security classification to be \emph{DevOps-ready}, two of which will be the focus for the rest of the paper, namely to be tool-based and easy to use for non-security experts, such as ordinary developers or system architects. We then exemplify how one can make a security classification methodology DevOps-ready. We do this through an interaction design process, where we create and evaluate the usability of a tool implementing the chosen methodology. Since such work seems to be new within the usable security community, and even more so in the software development (DevOps) community, we extract from our process a general, three-steps `recipe' that others can follow when making their own security methodologies DevOps-ready. The tool that we build is in itself a contribution of this process, as it can be independently used, extended, and/or integrated by developer teams into their DevSecOps tool-chains. Our tool is perceived (by the test subjects) as most useful in the design phase, but also during the testing phase where the security class would be one of the metrics used to evaluate the quality of their software.

Title: SegEarth-OV: Towards Traning-Free Open-Vocabulary Segmentation for Remote Sensing Images

• Authors: Kaiyu Li, Ruixun Liu, Xiangyong Cao, Deyu Meng, Zhi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01768
• Pdf URL: https://arxiv.org/pdf/2410.01768
• Copy Paste: [[2410.01768]] SegEarth-OV: Towards Traning-Free Open-Vocabulary Segmentation for Remote Sensing Images(https://arxiv.org/abs/2410.01768)
• Keywords: extraction, segmentation
• Abstract: Remote sensing image plays an irreplaceable role in fields such as agriculture, water resources, military, and disaster relief. Pixel-level interpretation is a critical aspect of remote sensing image applications; however, a prevalent limitation remains the need for extensive manual annotation. For this, we try to introduce open-vocabulary semantic segmentation (OVSS) into the remote sensing context. However, due to the sensitivity of remote sensing images to low-resolution features, distorted target shapes and ill-fitting boundaries are exhibited in the prediction mask. To tackle this issue, we propose a simple and general upsampler, SimFeatUp, to restore lost spatial information in deep features in a training-free style. Further, based on the observation of the abnormal response of local patch tokens to [CLS] token in CLIP, we propose to execute a straightforward subtraction operation to alleviate the global bias in patch tokens. Extensive experiments are conducted on 17 remote sensing datasets spanning semantic segmentation, building extraction, road detection, and flood detection tasks. Our method achieves an average of 5.8%, 8.2%, 4%, and 15.3% improvement over state-of-the-art methods on 4 tasks. All codes are released. \url{this https URL}

Title: Quantifying Generalization Complexity for Large Language Models

• Authors: Zhenting Qi, Hongyin Luo, Xuliang Huang, Zhuokai Zhao, Yibo Jiang, Xiangjun Fan, Himabindu Lakkaraju, James Glass
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01769
• Pdf URL: https://arxiv.org/pdf/2410.01769
• Copy Paste: [[2410.01769]] Quantifying Generalization Complexity for Large Language Models(https://arxiv.org/abs/2410.01769)
• Keywords: robust, large language model
• Abstract: While large language models (LLMs) have shown exceptional capabilities in understanding complex queries and performing sophisticated tasks, their generalization abilities are often deeply entangled with memorization, necessitating more precise evaluation. To address this challenge, we introduce Scylla, a dynamic evaluation framework that quantitatively measures the generalization abilities of LLMs. Scylla disentangles generalization from memorization via assessing model performance on both in-distribution (ID) and out-of-distribution (OOD) data through 20 tasks across 5 levels of complexity. Through extensive experiments, we uncover a non-monotonic relationship between task complexity and the performance gap between ID and OOD data, which we term the generalization valley. Specifically, this phenomenon reveals a critical threshold - referred to as critical complexity - where reliance on non-generalizable behavior peaks, indicating the upper bound of LLMs' generalization capabilities. As model size increases, the critical complexity shifts toward higher levels of task complexity, suggesting that larger models can handle more complex reasoning tasks before over-relying on memorization. Leveraging Scylla and the concept of critical complexity, we benchmark 28LLMs including both open-sourced models such as LLaMA and Qwen families, and close-sourced models like Claude and GPT, providing a more robust evaluation and establishing a clearer understanding of LLMs' generalization capabilities.

Title: Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context

• Authors: Spencer Frei, Gal Vardi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01774
• Pdf URL: https://arxiv.org/pdf/2410.01774
• Copy Paste: [[2410.01774]] Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context(https://arxiv.org/abs/2410.01774)
• Keywords: transformer
• Abstract: Transformers have the capacity to act as supervised learning algorithms: by properly encoding a set of labeled training ("in-context") examples and an unlabeled test example into an input sequence of vectors of the same dimension, the forward pass of the transformer can produce predictions for that unlabeled test example. A line of recent work has shown that when linear transformers are pre-trained on random instances for linear regression tasks, these trained transformers make predictions using an algorithm similar to that of ordinary least squares. In this work, we investigate the behavior of linear transformers trained on random linear classification tasks. Via an analysis of the implicit regularization of gradient descent, we characterize how many pre-training tasks and in-context examples are needed for the trained transformer to generalize well at test-time. We further show that in some settings, these trained transformers can exhibit "benign overfitting in-context": when in-context examples are corrupted by label flipping noise, the transformer memorizes all of its in-context examples (including those with noisy labels) yet still generalizes near-optimally for clean test examples.

Title: KeyVisor -- A Lightweight ISA Extension for Protected Key Handles with CPU-enforced Usage Policies

• Authors: Fabian Schwarz, Jan Philipp Thoma, Christian Rossow, Tim Güneysu
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2410.01777
• Pdf URL: https://arxiv.org/pdf/2410.01777
• Copy Paste: [[2410.01777]] KeyVisor -- A Lightweight ISA Extension for Protected Key Handles with CPU-enforced Usage Policies(https://arxiv.org/abs/2410.01777)
• Keywords: secure, security, protect, defense, attack, steal
• Abstract: The confidentiality of cryptographic keys is essential for the security of protection schemes used for communication, file encryption, and outsourced computation. Beyond cryptanalytic attacks, adversaries can steal keys from memory via software exploits or side channels, enabling them to, e.g., tamper with secrets or impersonate key owners. Therefore, existing defenses protect keys in dedicated devices or isolated memory, or store them only in encrypted form. However, these designs often provide unfavorable tradeoffs, sacrificing performance, fine-grained access control, or deployability. In this paper, we present KeyVisor, a lightweight ISA extension that securely offloads the handling of cryptographic keys to the CPU. KeyVisor provides CPU instructions that enable applications to request protected key handles and perform AEAD cipher operations on them. The underlying keys are accessible only by KeyVisor, and thus never leak to memory. KeyVisor's direct CPU integration enables fast crypto operations and hardware-enforced key usage restrictions, e.g., keys usable only for de-/encryption, with a limited lifetime, or with a process binding. Furthermore, privileged software, e.g., the monitor firmware of TEEs, can revoke keys or bind them to a specific process/TEE. We implement KeyVisor for RISC-V based on Rocket Chip, evaluate its performance, and demonstrate real-world use cases, including key-value databases, automotive feature licensing, and a read-only network middlebox.

Title: TopER: Topological Embeddings in Graph Representation Learning

• Authors: Astrit Tola, Funmilola Mary Taiwom, Cuneyt Gurcan Akcora, Baris Coskunuzer
• Subjects: cs.LG, math.AT
• Abstract URL: https://arxiv.org/abs/2410.01778
• Pdf URL: https://arxiv.org/pdf/2410.01778
• Copy Paste: [[2410.01778]] TopER: Topological Embeddings in Graph Representation Learning(https://arxiv.org/abs/2410.01778)
• Keywords: interpretability
• Abstract: Graph embeddings play a critical role in graph representation learning, allowing machine learning models to explore and interpret graph-structured data. However, existing methods often rely on opaque, high-dimensional embeddings, limiting interpretability and practical visualization. In this work, we introduce Topological Evolution Rate (TopER), a novel, low-dimensional embedding approach grounded in topological data analysis. TopER simplifies a key topological approach, Persistent Homology, by calculating the evolution rate of graph substructures, resulting in intuitive and interpretable visualizations of graph data. This approach not only enhances the exploration of graph datasets but also delivers competitive performance in graph clustering and classification tasks. Our TopER-based models achieve or surpass state-of-the-art results across molecular, biological, and social network datasets in tasks such as classification, clustering, and visualization.

Title: Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Models

• Authors: Shayekh Bin Islam, Md Asib Rahman, K S M Tozammel Hossain, Enamul Hoque, Shafiq Joty, Md Rizwan Parvez
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01782
• Pdf URL: https://arxiv.org/pdf/2410.01782
• Copy Paste: [[2410.01782]] Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Models(https://arxiv.org/abs/2410.01782)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) has been shown to enhance the factual accuracy of Large Language Models (LLMs), but existing methods often suffer from limited reasoning capabilities in effectively using the retrieved evidence, particularly when using open-source LLMs. To mitigate this gap, we introduce a novel framework, Open-RAG, designed to enhance reasoning capabilities in RAG with open-source LLMs. Our framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single- and multi-hop queries. Open-RAG uniquely trains the model to navigate challenging distractors that appear relevant but are misleading. As a result, Open-RAG leverages latent learning, dynamically selecting relevant experts and integrating external knowledge effectively for more accurate and contextually relevant responses. In addition, we propose a hybrid adaptive retrieval method to determine retrieval necessity and balance the trade-off between performance gain and inference speed. Experimental results show that the Llama2-7B-based Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT, Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source our code and models at this https URL

Title: Investigating on RLHF methodology

• Authors: Alexey Kutalev, Sergei Markoff
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01789
• Pdf URL: https://arxiv.org/pdf/2410.01789
• Copy Paste: [[2410.01789]] Investigating on RLHF methodology(https://arxiv.org/abs/2410.01789)
• Keywords: large language model
• Abstract: In this article, we investigate the alignment of Large Language Models according to human preferences. We discuss the features of training a Preference Model, which simulates human preferences, and the methods and details we found essential for achieving the best results. We also discuss using Reinforcement Learning to fine-tune Large Language Models and describe the challenges we faced and the ways to overcome them. Additionally, we present our experience with the Direct Preference Optimization method, which enables us to align a Large Language Model with human preferences without creating a separate Preference Model. As our contribution, we introduce the approach for collecting a preference dataset through perplexity filtering, which makes the process of creating such a dataset for a specific Language Model much easier and more cost-effective.

Title: When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1

• Authors: R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, Thomas L. Griffiths
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01792
• Pdf URL: https://arxiv.org/pdf/2410.01792
• Copy Paste: [[2410.01792]] When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1(https://arxiv.org/abs/2410.01792)
• Keywords: large language model
• Abstract: In "Embers of Autoregression" (McCoy et al., 2023), we showed that several large language models (LLMs) have some important limitations that are attributable to their origins in next-word prediction. Here we investigate whether these issues persist with o1, a new system from OpenAI that differs from previous LLMs in that it is optimized for reasoning. We find that o1 substantially outperforms previous LLMs in many cases, with particularly large improvements on rare variants of common tasks (e.g., forming acronyms from the second letter of each word in a list, rather than the first letter). Despite these quantitative improvements, however, o1 still displays the same qualitative trends that we observed in previous systems. Specifically, o1 - like previous LLMs - is sensitive to the probability of examples and tasks, performing better and requiring fewer "thinking tokens" in high-probability settings than in low-probability ones. These results show that optimizing a language model for reasoning can mitigate but might not fully overcome the language model's probability sensitivity.

Title: Loki: An Open-Source Tool for Fact Verification

• Authors: Haonan Li, Xudong Han, Hao Wang, Yuxia Wang, Minghan Wang, Rui Xing, Yilin Geng, Zenan Zhai, Preslav Nakov, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01794
• Pdf URL: https://arxiv.org/pdf/2410.01794
• Copy Paste: [[2410.01794]] Loki: An Open-Source Tool for Fact Verification(https://arxiv.org/abs/2410.01794)
• Keywords: robust
• Abstract: We introduce Loki, an open-source tool designed to address the growing problem of misinformation. Loki adopts a human-centered approach, striking a balance between the quality of fact-checking and the cost of human involvement. It decomposes the fact-checking task into a five-step pipeline: breaking down long texts into individual claims, assessing their check-worthiness, generating queries, retrieving evidence, and verifying the claims. Instead of fully automating the claim verification process, Loki provides essential information at each step to assist human judgment, especially for general users such as journalists and content moderators. Moreover, it has been optimized for latency, robustness, and cost efficiency at a commercially usable level. Loki is released under an MIT license and is available on GitHub. We also provide a video presenting the system and its capabilities.

Title: Knowledge-Driven Feature Selection and Engineering for Genotype Data with Large Language Models

• Authors: Joseph Lee, Shu Yang, Jae Young Baik, Xiaoxi Liu, Zhen Tan, Dawei Li, Zixuan Wen, Bojian Hou, Duy Duong-Tran, Tianlong Chen, Li Shen
• Subjects: cs.LG, cs.CL, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2410.01795
• Pdf URL: https://arxiv.org/pdf/2410.01795
• Copy Paste: [[2410.01795]] Knowledge-Driven Feature Selection and Engineering for Genotype Data with Large Language Models(https://arxiv.org/abs/2410.01795)
• Keywords: robust, large language model
• Abstract: Predicting phenotypes with complex genetic bases based on a small, interpretable set of variant features remains a challenging task. Conventionally, data-driven approaches are utilized for this task, yet the high dimensional nature of genotype data makes the analysis and prediction difficult. Motivated by the extensive knowledge encoded in pre-trained LLMs and their success in processing complex biomedical concepts, we set to examine the ability of LLMs in feature selection and engineering for tabular genotype data, with a novel knowledge-driven framework. We develop FREEFORM, Free-flow Reasoning and Ensembling for Enhanced Feature Output and Robust Modeling, designed with chain-of-thought and ensembling principles, to select and engineer features with the intrinsic knowledge of LLMs. Evaluated on two distinct genotype-phenotype datasets, genetic ancestry and hereditary hearing loss, we find this framework outperforms several data-driven methods, particularly on low-shot regimes. FREEFORM is available as open-source framework at GitHub: this https URL.

Title: Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space

• Authors: Yangming Li, Chieh-Hsin Lai, Carola-Bibiane Schönlieb, Yuki Mitsufuji, Stefano Ermon
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01796
• Pdf URL: https://arxiv.org/pdf/2410.01796
• Copy Paste: [[2410.01796]] Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space(https://arxiv.org/abs/2410.01796)
• Keywords: diffusion, generative
• Abstract: Deep Generative Models (DGMs), including Energy-Based Models (EBMs) and Score-based Generative Models (SGMs), have advanced high-fidelity data generation and complex continuous distribution approximation. However, their application in Markov Decision Processes (MDPs), particularly in distributional Reinforcement Learning (RL), remains underexplored, with conventional histogram-based methods dominating the field. This paper rigorously highlights that this application gap is caused by the nonlinearity of modern DGMs, which conflicts with the linearity required by the Bellman equation in MDPs. For instance, EBMs involve nonlinear operations such as exponentiating energy functions and normalizing constants. To address this, we introduce Bellman Diffusion, a novel DGM framework that maintains linearity in MDPs through gradient and scalar field modeling. With divergence-based training techniques to optimize neural network proxies and a new type of stochastic differential equation (SDE) for sampling, Bellman Diffusion is guaranteed to converge to the target distribution. Our empirical results show that Bellman Diffusion achieves accurate field estimations and is a capable image generator, converging 1.5x faster than the traditional histogram-based baseline in distributional RL tasks. This work enables the effective integration of DGMs into MDP applications, unlocking new avenues for advanced decision-making frameworks.

Title: FabricDiffusion: High-Fidelity Texture Transfer for 3D Garments Generation from In-The-Wild Clothing Images

• Authors: Cheng Zhang, Yuanhao Wang, Francisco Vicente Carrasco, Chenglei Wu, Jinlong Yang, Thabo Beeler, Fernando De la Torre
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.01801
• Pdf URL: https://arxiv.org/pdf/2410.01801
• Copy Paste: [[2410.01801]] FabricDiffusion: High-Fidelity Texture Transfer for 3D Garments Generation from In-The-Wild Clothing Images(https://arxiv.org/abs/2410.01801)
• Keywords: diffusion, generative
• Abstract: We introduce FabricDiffusion, a method for transferring fabric textures from a single clothing image to 3D garments of arbitrary shapes. Existing approaches typically synthesize textures on the garment surface through 2D-to-3D texture mapping or depth-aware inpainting via generative models. Unfortunately, these methods often struggle to capture and preserve texture details, particularly due to challenging occlusions, distortions, or poses in the input image. Inspired by the observation that in the fashion industry, most garments are constructed by stitching sewing patterns with flat, repeatable textures, we cast the task of clothing texture transfer as extracting distortion-free, tileable texture materials that are subsequently mapped onto the UV space of the garment. Building upon this insight, we train a denoising diffusion model with a large-scale synthetic dataset to rectify distortions in the input texture image. This process yields a flat texture map that enables a tight coupling with existing Physically-Based Rendering (PBR) material generation pipelines, allowing for realistic relighting of the garment under various lighting conditions. We show that FabricDiffusion can transfer various features from a single clothing image including texture patterns, material properties, and detailed prints and logos. Extensive experiments demonstrate that our model significantly outperforms state-to-the-art methods on both synthetic data and real-world, in-the-wild clothing images while generalizing to unseen textures and garment shapes.

Title: Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads

• Authors: Yuxiang Huang, Binhang Yuan, Xu Han, Chaojun Xiao, Zhiyuan Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01805
• Pdf URL: https://arxiv.org/pdf/2410.01805
• Copy Paste: [[2410.01805]] Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads(https://arxiv.org/abs/2410.01805)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) have shown remarkable advances in supporting long-context comprehension and processing tasks. However, scaling the generation inference of LLMs to such long contexts incurs significant additional computation load, and demands a substantial GPU memory footprint to maintain the key-value (KV) cache of transformer-based LLMs. Existing KV cache compression methods, such as quantization, face memory bottlenecks as context length increases, while static-sized caches, such as eviction, suffer from inefficient policies. These limitations restrict deployment on consumer-grade devices like a single Nvidia 4090 GPU. To overcome this, we propose Locret, a framework for long-context LLM inference that introduces retaining heads to evaluate the causal importance of KV cache units, allowing for more accurate eviction within a fixed cache size. Locret is fine-tuned on top of the frozen backbone LLM using a minimal amount of data from standard long-context SFT datasets. During inference, we evict low-importance cache units along with a chunked prefill pattern, significantly reducing peak GPU memory usage. We conduct an extensive empirical study to evaluate Locret, where the experimental results show that Locret outperforms the recent competitive approaches, including InfLLM, Quantization, SirLLM, and MInference, in terms of memory efficiency and the quality of generated contents -- Locret achieves over a 20x and 8x KV cache compression ratio compared to the full KV cache for Phi-3-mini-128K and Llama-3.1-8B-instruct. Additionally, Locret can be combined with other methods, such as quantization and token merging. To our knowledge, Locret is the first framework capable of deploying Llama-3.1-8B or similar models on a single Nvidia 4090 GPU, enabling 128K long-context inference without compromising generation quality, and requiring little additional system optimizations.