2024-10-11

Title: Neural Contrast: Leveraging Generative Editing for Graphic Design Recommendations

• Authors: Marian Lupascu, Ionut Mironica, Mihai-Sorin Stupariu
• Subjects: cs.CV, cs.GR, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07211
• Pdf URL: https://arxiv.org/pdf/2410.07211
• Copy Paste: [[2410.07211]] Neural Contrast: Leveraging Generative Editing for Graphic Design Recommendations(https://arxiv.org/abs/2410.07211)
• Keywords: diffusion, generative
• Abstract: Creating visually appealing composites requires optimizing both text and background for compatibility. Previous methods have focused on simple design strategies, such as changing text color or adding background shapes for contrast. These approaches are often destructive, altering text color or partially obstructing the background image. Another method involves placing design elements in non-salient and contrasting regions, but this isn't always effective, especially with patterned backgrounds. To address these challenges, we propose a generative approach using a diffusion model. This method ensures the altered regions beneath design assets exhibit low saliency while enhancing contrast, thereby improving the visibility of the design asset.

Title: BlockMEDC: Blockchain Smart Contracts for Securing Moroccan Higher Education Digital Certificates

• Authors: Mohamed Fartitchou, Ismail Lamaakal, Khalid El Makkaoui, Zakaria El Allali, Yassine Maleh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.07258
• Pdf URL: https://arxiv.org/pdf/2410.07258
• Copy Paste: [[2410.07258]] BlockMEDC: Blockchain Smart Contracts for Securing Moroccan Higher Education Digital Certificates(https://arxiv.org/abs/2410.07258)
• Keywords: secure
• Abstract: Morocco's Vision 2030, known as Maroc Digital 2030, aims to position the country as a regional leader in digital technology by boosting digital infrastructure, fostering innovation, and advancing digital skills. Complementing this initiative, the Pacte ESRI 2030 strategy, launched in 2023, seeks to transform the higher education, research, and innovation sectors by integrating state-of-the-art digital technologies. In alignment with these national strategies, this paper introduces BlockMEDC, a blockchain-based system for securing and managing Moroccan educational digital certificates. Leveraging Ethereum smart contracts and the InterPlanetary File System, BlockMEDC automates the issuance, management, and verification of academic credentials across Moroccan universities. The proposed system addresses key issues such as document authenticity, manual verification, and lack of interoperability, delivering a secure, transparent, and cost-effective solution that aligns with Morocco's digital transformation goals for the education sector.

Title: Memory-augmented Transformers can implement Linear First-Order Optimization Methods

• Authors: Sanchayan Dutta (UC Davis), Suvrit Sra (TU Munich)
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2410.07263
• Pdf URL: https://arxiv.org/pdf/2410.07263
• Copy Paste: [[2410.07263]] Memory-augmented Transformers can implement Linear First-Order Optimization Methods(https://arxiv.org/abs/2410.07263)
• Keywords: transformer
• Abstract: We show that memory-augmented Transformers (Memformers) can implement linear first-order optimization methods such as conjugate gradient descent, momentum methods, and more generally, methods that linearly combine past gradients. Building on prior work that demonstrates how Transformers can simulate preconditioned gradient descent, we provide theoretical and empirical evidence that Memformers can learn more advanced optimization algorithms. Specifically, we analyze how memory registers in Memformers store suitable intermediate attention values allowing them to implement algorithms such as conjugate gradient. Our results show that Memformers can efficiently learn these methods by training on random linear regression tasks, even learning methods that outperform conjugate gradient. This work extends our knowledge about the algorithmic capabilities of Transformers, showing how they can learn complex optimization methods.

Title: Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representation

• Authors: Yuxin Li, Yiheng Li, Xulei Yang, Mengying Yu, Zihang Huang, Xiaojun Wu, Chai Kiat Yeo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07268
• Pdf URL: https://arxiv.org/pdf/2410.07268
• Copy Paste: [[2410.07268]] Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representation(https://arxiv.org/abs/2410.07268)
• Keywords: extraction
• Abstract: In the landscape of autonomous driving, Bird's-Eye-View (BEV) representation has recently garnered substantial academic attention, serving as a transformative framework for the fusion of multi-modal sensor inputs. This BEV paradigm effectively shifts the sensor fusion challenge from a rule-based methodology to a data-centric approach, thereby facilitating more nuanced feature extraction from an array of heterogeneous sensors. Notwithstanding its evident merits, the computational overhead associated with BEV-based techniques often mandates high-capacity hardware infrastructures, thus posing challenges for practical, real-world implementations. To mitigate this limitation, we introduce a novel content-aware multi-modal joint input pruning technique. Our method leverages BEV as a shared anchor to algorithmically identify and eliminate non-essential sensor regions prior to their introduction into the perception model's backbone. We validatethe efficacy of our approach through extensive experiments on the NuScenes dataset, demonstrating substantial computational efficiency without sacrificing perception accuracy. To the best of our knowledge, this work represents the first attempt to alleviate the computational burden from the input pruning point.

Title: Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration

• Authors: Qinglun Li, Miao Zhang, Yingqi Liu, Quanjun Yin, Li Shen, Xiaochun Cao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07272
• Pdf URL: https://arxiv.org/pdf/2410.07272
• Copy Paste: [[2410.07272]] Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration(https://arxiv.org/abs/2410.07272)
• Keywords: privacy, federate
• Abstract: Decentralized Federated Learning has emerged as an alternative to centralized architectures due to its faster training, privacy preservation, and reduced communication overhead. In decentralized communication, the server aggregation phase in Centralized Federated Learning shifts to the client side, which means that clients connect with each other in a peer-to-peer manner. However, compared to the centralized mode, data heterogeneity in Decentralized Federated Learning will cause larger variances between aggregated models, which leads to slow convergence in training and poor generalization performance in tests. To address these issues, we introduce Catalyst Acceleration and propose an acceleration Decentralized Federated Learning algorithm called DFedCata. It consists of two main components: the Moreau envelope function, which primarily addresses parameter inconsistencies among clients caused by data heterogeneity, and Nesterov's extrapolation step, which accelerates the aggregation phase. Theoretically, we prove the optimization error bound and generalization error bound of the algorithm, providing a further understanding of the nature of the algorithm and the theoretical perspectives on the hyperparameter choice. Empirically, we demonstrate the advantages of the proposed algorithm in both convergence speed and generalization performance on CIFAR10/100 with various non-iid data distributions. Furthermore, we also experimentally verify the theoretical properties of DFedCata.

Title: BELM: Bidirectional Explicit Linear Multi-step Sampler for Exact Inversion in Diffusion Models

• Authors: Fangyikang Wang, Hubery Yin, Yuejiang Dong, Huminhao Zhu, Chao Zhang, Hanbin Zhao, Hui Qian, Chen Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07273
• Pdf URL: https://arxiv.org/pdf/2410.07273
• Copy Paste: [[2410.07273]] BELM: Bidirectional Explicit Linear Multi-step Sampler for Exact Inversion in Diffusion Models(https://arxiv.org/abs/2410.07273)
• Keywords: diffusion
• Abstract: The inversion of diffusion model sampling, which aims to find the corresponding initial noise of a sample, plays a critical role in various tasks. Recently, several heuristic exact inversion samplers have been proposed to address the inexact inversion issue in a training-free manner. However, the theoretical properties of these heuristic samplers remain unknown and they often exhibit mediocre sampling quality. In this paper, we introduce a generic formulation, \emph{Bidirectional Explicit Linear Multi-step} (BELM) samplers, of the exact inversion samplers, which includes all previously proposed heuristic exact inversion samplers as special cases. The BELM formulation is derived from the variable-stepsize-variable-formula linear multi-step method via integrating a bidirectional explicit constraint. We highlight this bidirectional explicit constraint is the key of mathematically exact inversion. We systematically investigate the Local Truncation Error (LTE) within the BELM framework and show that the existing heuristic designs of exact inversion samplers yield sub-optimal LTE. Consequently, we propose the Optimal BELM (O-BELM) sampler through the LTE minimization approach. We conduct additional analysis to substantiate the theoretical stability and global convergence property of the proposed optimal sampler. Comprehensive experiments demonstrate our O-BELM sampler establishes the exact inversion property while achieving high-quality sampling. Additional experiments in image editing and image interpolation highlight the extensive potential of applying O-BELM in varying applications.

Title: Mitigation of gender bias in automatic facial non-verbal behaviors generation

• Authors: Alice Delbosc (TALEP, LIS, AMU), Magalie Ochs (LIS, AMU, R2I), Nicolas Sabouret (CPU, LISN), Brian Ravenet (CPU, LISN), Stephane Ayache (AMU, LIS, QARMA)
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2410.07274
• Pdf URL: https://arxiv.org/pdf/2410.07274
• Copy Paste: [[2410.07274]] Mitigation of gender bias in automatic facial non-verbal behaviors generation(https://arxiv.org/abs/2410.07274)
• Keywords: fair
• Abstract: Research on non-verbal behavior generation for social interactive agents focuses mainly on the believability and synchronization of non-verbal cues with speech. However, existing models, predominantly based on deep learning architectures, often perpetuate biases inherent in the training data. This raises ethical concerns, depending on the intended application of these agents. This paper addresses these issues by first examining the influence of gender on facial non-verbal behaviors. We concentrate on gaze, head movements, and facial expressions. We introduce a classifier capable of discerning the gender of a speaker from their non-verbal cues. This classifier achieves high accuracy on both real behavior data, extracted using state-of-the-art tools, and synthetic data, generated from a model developed in previous this http URL upon this work, we present a new model, FairGenderGen, which integrates a gender discriminator and a gradient reversal layer into our previous behavior generation model. This new model generates facial non-verbal behaviors from speech features, mitigating gender sensitivity in the generated behaviors. Our experiments demonstrate that the classifier, developed in the initial phase, is no longer effective in distinguishing the gender of the speaker from the generated non-verbal behaviors.

Title: Retrieval Replace Reduction: An effective visual token reduction method via semantic match

• Authors: Yingen Liu, Fan Wu, Ruihui Li, Zhuo Tang, Kenli Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07278
• Pdf URL: https://arxiv.org/pdf/2410.07278
• Copy Paste: [[2410.07278]] Retrieval Replace Reduction: An effective visual token reduction method via semantic match(https://arxiv.org/abs/2410.07278)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) have demonstrated strong performance across various tasks without requiring training from scratch. However, they face significant computational and memory constraints, particularly when processing multimodal inputs that exceed context length, limiting their scalability. In this paper, we introduce a new approach, \textbf{TRSM} (\textbf{T}oken \textbf{R}eduction via \textbf{S}emantic \textbf{M}atch), which effectively reduces the number of visual tokens without compromising MLLM performance. Inspired by how humans process multimodal tasks, TRSM leverages semantic information from one modality to match relevant semantics in another, reducing the number of visual this http URL, to retain task relevant visual tokens, we use the text prompt as a query vector to retrieve the most similar vectors from the visual prompt and merge them with the text tokens. Based on experimental results, when applied to LLaVA-1.5\cite{liu2023}, our approach compresses the visual tokens by 20\%, achieving comparable performance across diverse visual question-answering and reasoning tasks.

Title: Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning

• Authors: Zhilong Li, Xiaohu Wu, Xiaoli Tang, Tiantian He, Yew-Soon Ong, Mengmeng Chen, Qiqi Liu, Qicheng Lao, Xiaoxiao Li, Han Yu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07286
• Pdf URL: https://arxiv.org/pdf/2410.07286
• Copy Paste: [[2410.07286]] Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning(https://arxiv.org/abs/2410.07286)
• Keywords: federate, fair
• Abstract: There is growing research interest in measuring the statistical heterogeneity of clients' local datasets. Such measurements are used to estimate the suitability for collaborative training of personalized federated learning (PFL) models. Currently, these research endeavors are taking place in silos and there is a lack of a unified benchmark to provide a fair and convenient comparison among various approaches in common settings. We aim to bridge this important gap in this paper. The proposed benchmarking framework currently includes six representative approaches. Extensive experiments have been conducted to compare these approaches under five standard non-IID FL settings, providing much needed insights into which approaches are advantageous under which settings. The proposed framework offers useful guidance on the suitability of various data divergence measures in FL systems. It is beneficial for keeping related research activities on the right track in terms of: (1) designing PFL schemes, (2) selecting appropriate data heterogeneity evaluation approaches for specific FL application scenarios, and (3) addressing fairness issues in collaborative model training. The code is available at this https URL.

Title: ReinDiffuse: Crafting Physically Plausible Motions with Reinforced Diffusion Model

• Authors: Gaoge Han, Mingjiang Liang, Jinglei Tang, Yongkang Cheng, Wei Liu, Shaoli Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07296
• Pdf URL: https://arxiv.org/pdf/2410.07296
• Copy Paste: [[2410.07296]] ReinDiffuse: Crafting Physically Plausible Motions with Reinforced Diffusion Model(https://arxiv.org/abs/2410.07296)
• Keywords: diffusion
• Abstract: Generating human motion from textual descriptions is a challenging task. Existing methods either struggle with physical credibility or are limited by the complexities of physics simulations. In this paper, we present \emph{ReinDiffuse} that combines reinforcement learning with motion diffusion model to generate physically credible human motions that align with textual descriptions. Our method adapts Motion Diffusion Model to output a parameterized distribution of actions, making them compatible with reinforcement learning paradigms. We employ reinforcement learning with the objective of maximizing physically plausible rewards to optimize motion generation for physical fidelity. Our approach outperforms existing state-of-the-art models on two major datasets, HumanML3D and KIT-ML, achieving significant improvements in physical plausibility and motion quality. Project: \url{this https URL}

Title: Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow

• Authors: Fu-Yun Wang, Ling Yang, Zhaoyang Huang, Mengdi Wang, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07303
• Pdf URL: https://arxiv.org/pdf/2410.07303
• Copy Paste: [[2410.07303]] Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow(https://arxiv.org/abs/2410.07303)
• Keywords: diffusion, generative
• Abstract: Diffusion models have greatly improved visual generation but are hindered by slow generation speed due to the computationally intensive nature of solving generative ODEs. Rectified flow, a widely recognized solution, improves generation speed by straightening the ODE path. Its key components include: 1) using the diffusion form of flow-matching, 2) employing $\boldsymbol v$-prediction, and 3) performing rectification (a.k.a. reflow). In this paper, we argue that the success of rectification primarily lies in using a pretrained diffusion model to obtain matched pairs of noise and samples, followed by retraining with these matched noise-sample pairs. Based on this, components 1) and 2) are unnecessary. Furthermore, we highlight that straightness is not an essential training target for rectification; rather, it is a specific case of flow-matching models. The more critical training target is to achieve a first-order approximate ODE path, which is inherently curved for models like DDPM and Sub-VP. Building on this insight, we propose Rectified Diffusion, which generalizes the design space and application scope of rectification to encompass the broader category of diffusion models, rather than being restricted to flow-matching models. We validate our method on Stable Diffusion v1-5 and Stable Diffusion XL. Our method not only greatly simplifies the training procedure of rectified flow-based previous works (e.g., InstaFlow) but also achieves superior performance with even lower training cost. Our code is available at this https URL.

Title: DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models

• Authors: Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, Kang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07331
• Pdf URL: https://arxiv.org/pdf/2410.07331
• Copy Paste: [[2410.07331]] DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models(https://arxiv.org/abs/2410.07331)
• Keywords: robust, large language model
• Abstract: We introduce DA-Code, a code generation benchmark specifically designed to assess LLMs on agent-based data science tasks. This benchmark features three core elements: First, the tasks within DA-Code are inherently challenging, setting them apart from traditional code generation tasks and demanding advanced coding skills in grounding and planning. Second, examples in DA-Code are all based on real and diverse data, covering a wide range of complex data wrangling and analytics tasks. Third, to solve the tasks, the models must utilize complex data science programming languages, to perform intricate data processing and derive the answers. We set up the benchmark in a controllable and executable environment that aligns with real-world data analysis scenarios and is scalable. The annotators meticulously design the evaluation suite to ensure the accuracy and robustness of the evaluation. We develop the DA-Agent baseline. Experiments show that although the baseline performs better than other existing frameworks, using the current best LLMs achieves only 30.5% accuracy, leaving ample room for improvement. We release our benchmark at [this https URL](this https URL).

Title: An undetectable watermark for generative image models

• Authors: Sam Gunn, Xuandong Zhao, Dawn Song
• Subjects: cs.CR, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.07369
• Pdf URL: https://arxiv.org/pdf/2410.07369
• Copy Paste: [[2410.07369]] An undetectable watermark for generative image models(https://arxiv.org/abs/2410.07369)
• Keywords: attack, robust, watermark, diffusion, generative
• Abstract: We present the first undetectable watermarking scheme for generative image models. Undetectability ensures that no efficient adversary can distinguish between watermarked and un-watermarked images, even after making many adaptive queries. In particular, an undetectable watermark does not degrade image quality under any efficiently computable metric. Our scheme works by selecting the initial latents of a diffusion model using a pseudorandom error-correcting code (Christ and Gunn, 2024), a strategy which guarantees undetectability and robustness. We experimentally demonstrate that our watermarks are quality-preserving and robust using Stable Diffusion 2.1. Our experiments verify that, in contrast to every prior scheme we tested, our watermark does not degrade image quality. Our experiments also demonstrate robustness: existing watermark removal attacks fail to remove our watermark from images without significantly degrading the quality of the images. Finally, we find that we can robustly encode 512 bits in our watermark, and up to 2500 bits when the images are not subjected to watermark removal attacks. Our code is available at this https URL.

Title: SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers

• Authors: Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Alexander Mikhalev, Alexander Panchenko, Ivan Oseledets
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07383
• Pdf URL: https://arxiv.org/pdf/2410.07383
• Copy Paste: [[2410.07383]] SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers(https://arxiv.org/abs/2410.07383)
• Keywords: robust, transformer
• Abstract: The performance of Transformer models has been enhanced by increasing the number of parameters and the length of the processed text. Consequently, fine-tuning the entire model becomes a memory-intensive process. High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and often overlook MLP blocks, which contain about half of the model parameters. We propose a new selective PEFT method, namely SparseGrad, that performs well on MLP blocks. We transfer layer gradients to a space where only about 1\% of the layer's elements remain significant. By converting gradients into a sparse structure, we reduce the number of updated parameters. We apply SparseGrad to fine-tune BERT and RoBERTa for the NLU task and LLaMa-2 for the Question-Answering task. In these experiments, with identical memory requirements, our method outperforms LoRA and MeProp, robust popular state-of-the-art PEFT approaches.

Title: LLM Embeddings Improve Test-time Adaptation to Tabular $Y|X$-Shifts

• Authors: Yibo Zeng, Jiashuo Liu, Henry Lam, Hongseok Namkoong
• Subjects: cs.LG, cs.AI, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07395
• Pdf URL: https://arxiv.org/pdf/2410.07395
• Copy Paste: [[2410.07395]] LLM Embeddings Improve Test-time Adaptation to Tabular $Y|X$-Shifts(https://arxiv.org/abs/2410.07395)
• Keywords: robust
• Abstract: For tabular datasets, the change in the relationship between the label and covariates ($Y|X$-shifts) is common due to missing variables (a.k.a. confounders). Since it is impossible to generalize to a completely new and unknown domain, we study models that are easy to adapt to the target domain even with few labeled examples. We focus on building more informative representations of tabular data that can mitigate $Y|X$-shifts, and propose to leverage the prior world knowledge in LLMs by serializing (write down) the tabular data to encode it. We find LLM embeddings alone provide inconsistent improvements in robustness, but models trained on them can be well adapted/finetuned to the target domain even using 32 labeled observations. Our finding is based on a comprehensive and systematic study consisting of 7650 source-target pairs and benchmark against 261,000 model configurations trained by 22 algorithms. Our observation holds when ablating the size of accessible target data and different adaptation strategies. The code is available at this https URL.

Title: Enhancing Soccer Camera Calibration Through Keypoint Exploitation

• Authors: Nikolay S. Falaleev, Ruilong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07401
• Pdf URL: https://arxiv.org/pdf/2410.07401
• Copy Paste: [[2410.07401]] Enhancing Soccer Camera Calibration Through Keypoint Exploitation(https://arxiv.org/abs/2410.07401)
• Keywords: secure
• Abstract: Accurate camera calibration is essential for transforming 2D images from camera sensors into 3D world coordinates, enabling precise scene geometry interpretation and supporting sports analytics tasks such as player tracking, offside detection, and performance analysis. However, obtaining a sufficient number of high-quality point pairs remains a significant challenge for both traditional and deep learning-based calibration methods. This paper introduces a multi-stage pipeline that addresses this challenge by leveraging the structural features of the football pitch. Our approach significantly increases the number of usable points for calibration by exploiting line-line and line-conic intersections, points on the conics, and other geometric features. To mitigate the impact of imperfect annotations, we employ data fitting techniques. Our pipeline utilizes deep learning for keypoint and line detection and incorporates geometric constraints based on real-world pitch dimensions. A voter algorithm iteratively selects the most reliable keypoints, further enhancing calibration accuracy. We evaluated our approach on the largest football broadcast camera calibration dataset available, and secured the top position in the SoccerNet Camera Calibration Challenge 2023 [arXiv:2309.06006], which demonstrates the effectiveness of our method in real-world scenarios. The project code is available at this https URL .

Title: Bayes-Nash Generative Privacy Protection Against Membership Inference Attacks

• Authors: Tao Zhang, Rajagopal Venkatesaraman, Rajat K. De, Bradley A. Malin, Yevgeniy Vorobeychik
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.07414
• Pdf URL: https://arxiv.org/pdf/2410.07414
• Copy Paste: [[2410.07414]] Bayes-Nash Generative Privacy Protection Against Membership Inference Attacks(https://arxiv.org/abs/2410.07414)
• Keywords: privacy, protect, attack, robust, membership infer, generative
• Abstract: An ability to share data, even in aggregated form, is critical to advancing both conventional and data science. However, insofar as such datasets are comprised of individuals, their membership in these datasets is often viewed as sensitive, with membership inference attacks (MIAs) threatening to violate their privacy. We propose a Bayesian game model for privacy-preserving publishing of data-sharing mechanism outputs (for example, summary statistics for sharing genomic data). In this game, the defender minimizes a combination of expected utility and privacy loss, with the latter being maximized by a Bayes-rational attacker. We propose a GAN-style algorithm to approximate a Bayes-Nash equilibrium of this game, and introduce the notions of Bayes-Nash generative privacy (BNGP) and Bayes generative privacy (BGP) risk that aims to optimally balance the defender's privacy and utility in a way that is robust to the attacker's heterogeneous preferences with respect to true and false positives. We demonstrate the properties of composition and post-processing for BGP risk and establish conditions under which BNGP and pure differential privacy (PDP) are equivalent. We apply our method to sharing summary statistics, where MIAs can re-identify individuals even from aggregated data. Theoretical analysis and empirical results demonstrate that our Bayesian game-theoretic method outperforms state-of-the-art approaches for privacy-preserving sharing of summary statistics.

Title: 3D2M Dataset: A 3-Dimension diverse Mesh Dataset

• Authors: Sankarshan Dasgupta
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.07415
• Pdf URL: https://arxiv.org/pdf/2410.07415
• Copy Paste: [[2410.07415]] 3D2M Dataset: A 3-Dimension diverse Mesh Dataset(https://arxiv.org/abs/2410.07415)
• Keywords: robust
• Abstract: Three-dimensional (3D) reconstruction has emerged as a prominent area of research, attracting significant attention from academia and industry alike. Among the various applications of 3D reconstruction, facial reconstruction poses some of the most formidable challenges. Additionally, each individuals facial structure is unique, requiring algorithms to be robust enough to handle this variability while maintaining fidelity to the original features. This article presents a comprehensive dataset of 3D meshes featuring a diverse range of facial structures and corresponding facial landmarks. The dataset comprises 188 3D facial meshes, including 73 from female candidates and 114 from male candidates. It encompasses a broad representation of ethnic backgrounds, with contributions from 45 different ethnicities, ensuring a rich diversity in facial characteristics. Each facial mesh is accompanied by key points that accurately annotate the relevant features, facilitating precise analysis and manipulation. This dataset is particularly valuable for applications such as facial re targeting, the study of facial structure components, and real-time person representation in video streams. By providing a robust resource for researchers and developers, it aims to advance the field of 3D facial reconstruction and related technologies.

Title: Segmenting objects with Bayesian fusion of active contour models and convnet priors

• Authors: Przemyslaw Polewski, Jacquelyn Shelton, Wei Yao, Marco Heurich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07421
• Pdf URL: https://arxiv.org/pdf/2410.07421
• Copy Paste: [[2410.07421]] Segmenting objects with Bayesian fusion of active contour models and convnet priors(https://arxiv.org/abs/2410.07421)
• Keywords: generative, segmentation
• Abstract: Instance segmentation is a core computer vision task with great practical significance. Recent advances, driven by large-scale benchmark datasets, have yielded good general-purpose Convolutional Neural Network (CNN)-based methods. Natural Resource Monitoring (NRM) utilizes remote sensing imagery with generally known scale and containing multiple overlapping instances of the same class, wherein the object contours are jagged and highly irregular. This is in stark contrast with the regular man-made objects found in classic benchmark datasets. We address this problem and propose a novel instance segmentation method geared towards NRM imagery. We formulate the problem as Bayesian maximum a posteriori inference which, in learning the individual object contours, incorporates shape, location, and position priors from state-of-the-art CNN architectures, driving a simultaneous level-set evolution of multiple object contours. We employ loose coupling between the CNNs that supply the priors and the active contour process, allowing a drop-in replacement of new network architectures. Moreover, we introduce a novel prior for contour shape, namely, a class of Deep Shape Models based on architectures from Generative Adversarial Networks (GANs). These Deep Shape Models are in essence a non-linear generalization of the classic Eigenshape formulation. In experiments, we tackle the challenging, real-world problem of segmenting individual dead tree crowns and delineating precise contours. We compare our method to two leading general-purpose instance segmentation methods - Mask R-CNN and K-net - on color infrared aerial imagery. Results show our approach to significantly outperform both methods in terms of reconstruction quality of tree crown contours. Furthermore, use of the GAN-based deep shape model prior yields significant improvement of all results over the vanilla Eigenshape prior.

Title: EventFlow: Forecasting Continuous-Time Event Data with Flow Matching

• Authors: Gavin Kerrigan, Kai Nelson, Padhraic Smyth
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07430
• Pdf URL: https://arxiv.org/pdf/2410.07430
• Copy Paste: [[2410.07430]] EventFlow: Forecasting Continuous-Time Event Data with Flow Matching(https://arxiv.org/abs/2410.07430)
• Keywords: generative
• Abstract: Continuous-time event sequences, in which events occur at irregular intervals, are ubiquitous across a wide range of industrial and scientific domains. The contemporary modeling paradigm is to treat such data as realizations of a temporal point process, and in machine learning it is common to model temporal point processes in an autoregressive fashion using a neural network. While autoregressive models are successful in predicting the time of a single subsequent event, their performance can be unsatisfactory in forecasting longer horizons due to cascading errors. We propose EventFlow, a non-autoregressive generative model for temporal point processes. Our model builds on the flow matching framework in order to directly learn joint distributions over event times, side-stepping the autoregressive process. EventFlow is likelihood-free, easy to implement and sample from, and either matches or surpasses the performance of state-of-the-art models in both unconditional and conditional generation tasks on a set of standard benchmarks

Title: Can Transformers Reason Logically? A Study in SAT Solving

• Authors: Leyan Pan, Vijay Ganesh, Jacob Abernethy, Chris Esposo, Wenke Lee
• Subjects: cs.LG, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2410.07432
• Pdf URL: https://arxiv.org/pdf/2410.07432
• Copy Paste: [[2410.07432]] Can Transformers Reason Logically? A Study in SAT Solving(https://arxiv.org/abs/2410.07432)
• Keywords: transformer
• Abstract: We theoretically and empirically study the logical reasoning capabilities of LLMs in the context of the Boolean satisfiability (SAT) problem. First, we construct a decoder-only Transformer that can solve SAT using backtracking and deduction via Chain-of-Thought (CoT). We prove its correctness by showing trace equivalence to the well-known DPLL SAT-solving algorithm. Second, to support the implementation of this abstract construction, we design a compiler $\texttt{PARAT}$ that takes as input a procedural specification and outputs a transformer model implementing this specification. Third, rather than $\textit{programming}$ a transformer to reason, we evaluate empirically whether it can be $\textit{trained}$ to do so by learning directly from algorithmic traces ("reasoning paths") of the DPLL algorithm.

Title: Toward Robust Real-World Audio Deepfake Detection: Closing the Explainability Gap

• Authors: Georgia Channing, Juil Sock, Ronald Clark, Philip Torr, Christian Schroeder de Witt
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.07436
• Pdf URL: https://arxiv.org/pdf/2410.07436
• Copy Paste: [[2410.07436]] Toward Robust Real-World Audio Deepfake Detection: Closing the Explainability Gap(https://arxiv.org/abs/2410.07436)
• Keywords: security, robust, explainability, transformer
• Abstract: The rapid proliferation of AI-manipulated or generated audio deepfakes poses serious challenges to media integrity and election security. Current AI-driven detection solutions lack explainability and underperform in real-world settings. In this paper, we introduce novel explainability methods for state-of-the-art transformer-based audio deepfake detectors and open-source a novel benchmark for real-world generalizability. By narrowing the explainability gap between transformer-based audio deepfake detectors and traditional methods, our results not only build trust with human experts, but also pave the way for unlocking the potential of citizen intelligence to overcome the scalability issue in audio deepfake detection.

Title: Robust infrared small target detection using self-supervised and a contrario paradigms

• Authors: Alina Ciocarlan, Sylvie Le Hégarat-Mascle, Sidonie Lefebvre, Arnaud Woiselle
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07437
• Pdf URL: https://arxiv.org/pdf/2410.07437
• Copy Paste: [[2410.07437]] Robust infrared small target detection using self-supervised and a contrario paradigms(https://arxiv.org/abs/2410.07437)
• Keywords: defense, robust, segmentation
• Abstract: Detecting small targets in infrared images poses significant challenges in defense applications due to the presence of complex backgrounds and the small size of the targets. Traditional object detection methods often struggle to balance high detection rates with low false alarm rates, especially when dealing with small objects. In this paper, we introduce a novel approach that combines a contrario paradigm with Self-Supervised Learning (SSL) to improve Infrared Small Target Detection (IRSTD). On the one hand, the integration of an a contrario criterion into a YOLO detection head enhances feature map responses for small and unexpected objects while effectively controlling false alarms. On the other hand, we explore SSL techniques to overcome the challenges of limited annotated data, common in IRSTD tasks. Specifically, we benchmark several representative SSL strategies for their effectiveness in improving small object detection performance. Our findings show that instance discrimination methods outperform masked image modeling strategies when applied to YOLO-based small object detection. Moreover, the combination of the a contrario and SSL paradigms leads to significant performance improvements, narrowing the gap with state-of-the-art segmentation methods and even outperforming them in frugal settings. This two-pronged approach offers a robust solution for improving IRSTD performance, particularly under challenging conditions.

Title: KACQ-DCNN: Uncertainty-Aware Interpretable Kolmogorov-Arnold Classical-Quantum Dual-Channel Neural Network for Heart Disease Detection

• Authors: Md Abrar Jahin, Md. Akmol Masud, M. F. Mridha, Zeyar Aung, Nilanjan Dey
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07446
• Pdf URL: https://arxiv.org/pdf/2410.07446
• Copy Paste: [[2410.07446]] KACQ-DCNN: Uncertainty-Aware Interpretable Kolmogorov-Arnold Classical-Quantum Dual-Channel Neural Network for Heart Disease Detection(https://arxiv.org/abs/2410.07446)
• Keywords: robust, explainability
• Abstract: Heart failure remains a major global health challenge, contributing significantly to the 17.8 million annual deaths from cardiovascular disease, highlighting the need for improved diagnostic tools. Current heart disease prediction models based on classical machine learning face limitations, including poor handling of high-dimensional, imbalanced data, limited performance on small datasets, and a lack of uncertainty quantification, while also being difficult for healthcare professionals to interpret. To address these issues, we introduce KACQ-DCNN, a novel classical-quantum hybrid dual-channel neural network that replaces traditional multilayer perceptrons and convolutional layers with Kolmogorov-Arnold Networks (KANs). This approach enhances function approximation with learnable univariate activation functions, reducing model complexity and improving generalization. The KACQ-DCNN 4-qubit 1-layered model significantly outperforms 37 benchmark models across multiple metrics, achieving an accuracy of 92.03%, a macro-average precision, recall, and F1 score of 92.00%, and an ROC-AUC score of 94.77%. Ablation studies demonstrate the synergistic benefits of combining classical and quantum components with KAN. Additionally, explainability techniques like LIME and SHAP provide feature-level insights, improving model transparency, while uncertainty quantification via conformal prediction ensures robust probability estimates. These results suggest that KACQ-DCNN offers a promising path toward more accurate, interpretable, and reliable heart disease predictions, paving the way for advancements in cardiovascular healthcare.

Title: Collective variables of neural networks: empirical time evolution and scaling laws

• Authors: Samuel Tovey, Sven Krippendorf, Michael Spannowsky, Konstantin Nikolaou, Christian Holm
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2410.07451
• Pdf URL: https://arxiv.org/pdf/2410.07451
• Copy Paste: [[2410.07451]] Collective variables of neural networks: empirical time evolution and scaling laws(https://arxiv.org/abs/2410.07451)
• Keywords: transformer
• Abstract: This work presents a novel means for understanding learning dynamics and scaling relations in neural networks. We show that certain measures on the spectrum of the empirical neural tangent kernel, specifically entropy and trace, yield insight into the representations learned by a neural network and how these can be improved through architecture scaling. These results are demonstrated first on test cases before being shown on more complex networks, including transformers, auto-encoders, graph neural networks, and reinforcement learning studies. In testing on a wide range of architectures, we highlight the universal nature of training dynamics and further discuss how it can be used to understand the mechanisms behind learning in neural networks. We identify two such dominant mechanisms present throughout machine learning training. The first, information compression, is seen through a reduction in the entropy of the NTK spectrum during training, and occurs predominantly in small neural networks. The second, coined structure formation, is seen through an increasing entropy and thus, the creation of structure in the neural network representations beyond the prior established by the network at initialization. Due to the ubiquity of the latter in deep neural network architectures and its flexibility in the creation of feature-rich representations, we argue that this form of evolution of the network's entropy be considered the onset of a deep learning regime.

Title: SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders

• Authors: Constantin Venhoff, Anisoara Calinescu, Philip Torr, Christian Schroeder de Witt
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07456
• Pdf URL: https://arxiv.org/pdf/2410.07456
• Copy Paste: [[2410.07456]] SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders(https://arxiv.org/abs/2410.07456)
• Keywords: interpretability
• Abstract: A key challenge in interpretability is to decompose model activations into meaningful features. Sparse autoencoders (SAEs) have emerged as a promising tool for this task. However, a central problem in evaluating the quality of SAEs is the absence of ground truth features to serve as an evaluation gold standard. Current evaluation methods for SAEs are therefore confronted with a significant trade-off: SAEs can either leverage toy models or other proxies with predefined ground truth features; or they use extensive prior knowledge of realistic task circuits. The former limits the generalizability of the evaluation results, while the latter limits the range of models and tasks that can be used for evaluations. We introduce SAGE: Scalable Autoencoder Ground-truth Evaluation, a ground truth evaluation framework for SAEs that scales to large state-of-the-art SAEs and models. We demonstrate that our method can automatically identify task-specific activations and compute ground truth features at these points. Compared to previous methods we reduce the training overhead by introducing a novel reconstruction method that allows to apply residual stream SAEs to sublayer activations. This eliminates the need for SAEs trained on every task-specific activation location. Then we validate the scalability of our framework, by evaluating SAEs on novel tasks on Pythia70M, GPT-2 Small, and Gemma-2-2. Our framework therefore paves the way for generalizable, large-scale evaluations of SAEs in interpretability research.

Title: Generalizing Segmentation Foundation Model Under Sim-to-real Domain-shift for Guidewire Segmentation in X-ray Fluoroscopy

• Authors: Yuxuan Wen, Evgenia Roussinova, Olivier Brina, Paolo Machi, Mohamed Bouri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07460
• Pdf URL: https://arxiv.org/pdf/2410.07460
• Copy Paste: [[2410.07460]] Generalizing Segmentation Foundation Model Under Sim-to-real Domain-shift for Guidewire Segmentation in X-ray Fluoroscopy(https://arxiv.org/abs/2410.07460)
• Keywords: segmentation
• Abstract: Guidewire segmentation during endovascular interventions holds the potential to significantly enhance procedural accuracy, improving visualization and providing critical feedback that can support both physicians and robotic systems in navigating complex vascular pathways. Unlike supervised segmentation networks, which need many expensive expert-annotated labels, sim-to-real domain adaptation approaches utilize synthetic data from simulations, offering a cost-effective solution. The success of models like Segment-Anything (SAM) has driven advancements in image segmentation foundation models with strong zero/few-shot generalization through prompt engineering. However, they struggle with medical images like X-ray fluoroscopy and the domain-shifts of the data. Given the challenges of acquiring annotation and the accessibility of labeled simulation data, we propose a sim-to-real domain adaption framework with a coarse-to-fine strategy to adapt SAM to X-ray fluoroscopy guidewire segmentation without any annotation on the target domain. We first generate the pseudo-labels by utilizing a simple source image style transfer technique that preserves the guidewire structure. Then, we develop a weakly supervised self-training architecture to fine-tune an end-to-end student SAM with the coarse labels by imposing consistency regularization and supervision from the teacher SAM network. We validate the effectiveness of the proposed method on a publicly available Cardiac dataset and an in-house Neurovascular dataset, where our method surpasses both pre-trained SAM and many state-of-the-art domain adaptation techniques by a large margin. Our code will be made public on GitHub soon.

Title: Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation

• Authors: Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07463
• Pdf URL: https://arxiv.org/pdf/2410.07463
• Copy Paste: [[2410.07463]] Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation(https://arxiv.org/abs/2410.07463)
• Keywords: diffusion, generative
• Abstract: In this paper, we introduce a novel task called language-guided joint audio-visual editing. Given an audio and image pair of a sounding event, this task aims at generating new audio-visual content by editing the given sounding event conditioned on the language guidance. For instance, we can alter the background environment of a sounding object while keeping its appearance unchanged, or we can add new sounds contextualized to the visual content. To address this task, we propose a new diffusion-based framework for joint audio-visual editing and introduce two key ideas. Firstly, we propose a one-shot adaptation approach to tailor generative diffusion models for audio-visual content editing. With as few as one audio-visual sample, we jointly transfer the audio and vision diffusion models to the target domain. After fine-tuning, our model enables consistent generation of this audio-visual sample. Secondly, we introduce a cross-modal semantic enhancement approach. We observe that when using language as content editing guidance, the vision branch may overlook editing requirements. This phenomenon, termed catastrophic neglect, hampers audio-visual alignment during content editing. We therefore enhance semantic consistency between language and vision to mitigate this issue. Extensive experiments validate the effectiveness of our method in language-based audio-visual editing and highlight its superiority over several baseline approaches. We recommend that readers visit our project page for more details: this https URL.

Title: SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

• Authors: Han Shen, Pin-Yu Chen, Payel Das, Tianyi Chen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07471
• Pdf URL: https://arxiv.org/pdf/2410.07471
• Copy Paste: [[2410.07471]] SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection(https://arxiv.org/abs/2410.07471)
• Keywords: large language model
• Abstract: Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs). However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5% and 9.7% win rate increase compared to random selection respectively on Llama-3-8b-Instruct and Merlinite-7b models. Our code is available on github this https URL.

Title: Exploring the design space of deep-learning-based weather forecasting systems

• Authors: Shoaib Ahmed Siddiqui, Jean Kossaifi, Boris Bonev, Christopher Choy, Jan Kautz, David Krueger, Kamyar Azizzadenesheli
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07472
• Pdf URL: https://arxiv.org/pdf/2410.07472
• Copy Paste: [[2410.07472]] Exploring the design space of deep-learning-based weather forecasting systems(https://arxiv.org/abs/2410.07472)
• Keywords: transformer
• Abstract: Despite tremendous progress in developing deep-learning-based weather forecasting systems, their design space, including the impact of different design choices, is yet to be well understood. This paper aims to fill this knowledge gap by systematically analyzing these choices including architecture, problem formulation, pretraining scheme, use of image-based pretrained models, loss functions, noise injection, multi-step inputs, additional static masks, multi-step finetuning (including larger stride models), as well as training on a larger dataset. We study fixed-grid architectures such as UNet, fully convolutional architectures, and transformer-based models, along with grid-invariant architectures, including graph-based and operator-based models. Our results show that fixed-grid architectures outperform grid-invariant architectures, indicating a need for further architectural developments in grid-invariant models such as neural operators. We therefore propose a hybrid system that combines the strong performance of fixed-grid models with the flexibility of grid-invariant architectures. We further show that multi-step fine-tuning is essential for most deep-learning models to work well in practice, which has been a common practice in the past. Pretraining objectives degrade performance in comparison to supervised training, while image-based pretrained models provide useful inductive biases in some cases in comparison to training the model from scratch. Interestingly, we see a strong positive effect of using a larger dataset when training a smaller model as compared to training on a smaller dataset for longer. Larger models, on the other hand, primarily benefit from just an increase in the computational budget. We believe that these results will aid in the design of better weather forecasting systems in the future.

Title: Progressive Multi-Modal Fusion for Robust 3D Object Detection

• Authors: Rohit Mohan, Daniele Cattaneo, Florian Drews, Abhinav Valada
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07475
• Pdf URL: https://arxiv.org/pdf/2410.07475
• Copy Paste: [[2410.07475]] Progressive Multi-Modal Fusion for Robust 3D Object Detection(https://arxiv.org/abs/2410.07475)
• Keywords: robust
• Abstract: Multi-sensor fusion is crucial for accurate 3D object detection in autonomous driving, with cameras and LiDAR being the most commonly used sensors. However, existing methods perform sensor fusion in a single view by projecting features from both modalities either in Bird's Eye View (BEV) or Perspective View (PV), thus sacrificing complementary information such as height or geometric proportions. To address this limitation, we propose ProFusion3D, a progressive fusion framework that combines features in both BEV and PV at both intermediate and object query levels. Our architecture hierarchically fuses local and global features, enhancing the robustness of 3D object detection. Additionally, we introduce a self-supervised mask modeling pre-training strategy to improve multi-modal representation learning and data efficiency through three novel objectives. Extensive experiments on nuScenes and Argoverse2 datasets conclusively demonstrate the efficacy of ProFusion3D. Moreover, ProFusion3D is robust to sensor failure, demonstrating strong performance when only one modality is available.

Title: Unifying and Verifying Mechanistic Interpretations: A Case Study with Group Operations

• Authors: Wilson Wu, Louis Jaburi, Jacob Drori, Jason Gross
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07476
• Pdf URL: https://arxiv.org/pdf/2410.07476
• Copy Paste: [[2410.07476]] Unifying and Verifying Mechanistic Interpretations: A Case Study with Group Operations(https://arxiv.org/abs/2410.07476)
• Keywords: interpretability
• Abstract: A recent line of work in mechanistic interpretability has focused on reverse-engineering the computation performed by neural networks trained on the binary operation of finite groups. We investigate the internals of one-hidden-layer neural networks trained on this task, revealing previously unidentified structure and producing a more complete description of such models that unifies the explanations of previous works. Notably, these models approximate equivariance in each input argument. We verify that our explanation applies to a large fraction of networks trained on this task by translating it into a compact proof of model performance, a quantitative evaluation of model understanding. In particular, our explanation yields a guarantee of model accuracy that runs in 30% the time of brute force and gives a >=95% accuracy bound for 45% of the models we trained. We were unable to obtain nontrivial non-vacuous accuracy bounds using only explanations from previous works.

Title: MoDEM: Mixture of Domain Expert Models

• Authors: Toby Simonds, Kemal Kurniawan, Jey Han Lau
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07490
• Pdf URL: https://arxiv.org/pdf/2410.07490
• Copy Paste: [[2410.07490]] MoDEM: Mixture of Domain Expert Models(https://arxiv.org/abs/2410.07490)
• Keywords: large language model
• Abstract: We propose a novel approach to enhancing the performance and efficiency of large language models (LLMs) by combining domain prompt routing with domain-specialized models. We introduce a system that utilizes a BERT-based router to direct incoming prompts to the most appropriate domain expert model. These expert models are specifically tuned for domains such as health, mathematics and science. Our research demonstrates that this approach can significantly outperform general-purpose models of comparable size, leading to a superior performance-to-cost ratio across various benchmarks. The implications of this study suggest a potential paradigm shift in LLM development and deployment. Rather than focusing solely on creating increasingly large, general-purpose models, the future of AI may lie in developing ecosystems of smaller, highly specialized models coupled with sophisticated routing systems. This approach could lead to more efficient resource utilization, reduced computational costs, and superior overall performance.

Title: PublicHearingBR: A Brazilian Portuguese Dataset of Public Hearing Transcripts for Summarization of Long Documents

• Authors: Leandro Carísio Fernandes, Guilherme Zeferino Rodrigues Dobins, Roberto Lotufo, Jayr Alencar Pereira
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07495
• Pdf URL: https://arxiv.org/pdf/2410.07495
• Copy Paste: [[2410.07495]] PublicHearingBR: A Brazilian Portuguese Dataset of Public Hearing Transcripts for Summarization of Long Documents(https://arxiv.org/abs/2410.07495)
• Keywords: large language model
• Abstract: This paper introduces PublicHearingBR, a Brazilian Portuguese dataset designed for summarizing long documents. The dataset consists of transcripts of public hearings held by the Brazilian Chamber of Deputies, paired with news articles and structured summaries containing the individuals participating in the hearing and their statements or opinions. The dataset supports the development and evaluation of long document summarization systems in Portuguese. Our contributions include the dataset, a hybrid summarization system to establish a baseline for future studies, and a discussion on evaluation metrics for summarization involving large language models, addressing the challenge of hallucination in the generated summaries. As a result of this discussion, the dataset also provides annotated data that can be used in Natural Language Inference tasks in Portuguese.

Title: Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labels

• Authors: Zhizheng Liu, Joe Lin, Wayne Wu, Bolei Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07500
• Pdf URL: https://arxiv.org/pdf/2410.07500
• Copy Paste: [[2410.07500]] Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labels(https://arxiv.org/abs/2410.07500)
• Keywords: generative
• Abstract: Understanding and modeling pedestrian movements in the real world is crucial for applications like motion forecasting and scene simulation. Many factors influence pedestrian movements, such as scene context, individual characteristics, and goals, which are often ignored by the existing human generation methods. Web videos contain natural pedestrian behavior and rich motion context, but annotating them with pre-trained predictors leads to noisy labels. In this work, we propose learning diverse pedestrian movements from web videos. We first curate a large-scale dataset called CityWalkers that captures diverse real-world pedestrian movements in urban scenes. Then, based on CityWalkers, we propose a generative model called PedGen for diverse pedestrian movement generation. PedGen introduces automatic label filtering to remove the low-quality labels and a mask embedding to train with partial labels. It also contains a novel context encoder that lifts the 2D scene context to 3D and can incorporate various context factors in generating realistic pedestrian movements in urban scenes. Experiments show that PedGen outperforms existing baseline methods for pedestrian movement generation by learning from noisy labels and incorporating the context factors. In addition, PedGen achieves zero-shot generalization in both real-world and simulated environments. The code, model, and data will be made publicly available at this https URL .

Title: Inferring biological processes with intrinsic noise from cross-sectional data

• Authors: Suryanarayana Maddu, Victor Chardès, Michael. J. Shelley
• Subjects: cs.LG, physics.bio-ph, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.07501
• Pdf URL: https://arxiv.org/pdf/2410.07501
• Copy Paste: [[2410.07501]] Inferring biological processes with intrinsic noise from cross-sectional data(https://arxiv.org/abs/2410.07501)
• Keywords: diffusion
• Abstract: Inferring dynamical models from data continues to be a significant challenge in computational biology, especially given the stochastic nature of many biological processes. We explore a common scenario in omics, where statistically independent cross-sectional samples are available at a few time points, and the goal is to infer the underlying diffusion process that generated the data. Existing inference approaches often simplify or ignore noise intrinsic to the system, compromising accuracy for the sake of optimization ease. We circumvent this compromise by inferring the phase-space probability flow that shares the same time-dependent marginal distributions as the underlying stochastic process. Our approach, probability flow inference (PFI), disentangles force from intrinsic stochasticity while retaining the algorithmic ease of ODE inference. Analytically, we prove that for Ornstein-Uhlenbeck processes the regularized PFI formalism yields a unique solution in the limit of well-sampled distributions. In practical applications, we show that PFI enables accurate parameter and force estimation in high-dimensional stochastic reaction networks, and that it allows inference of cell differentiation dynamics with molecular noise, outperforming state-of-the-art approaches.

Title: Adaptive Batch Size for Privately Finding Second-Order Stationary Points

• Authors: Daogao Liu, Kunal Talwar
• Subjects: cs.LG, cs.CR, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07502
• Pdf URL: https://arxiv.org/pdf/2410.07502
• Copy Paste: [[2410.07502]] Adaptive Batch Size for Privately Finding Second-Order Stationary Points(https://arxiv.org/abs/2410.07502)
• Keywords: privacy
• Abstract: There is a gap between finding a first-order stationary point (FOSP) and a second-order stationary point (SOSP) under differential privacy constraints, and it remains unclear whether privately finding an SOSP is more challenging than finding an FOSP. Specifically, Ganesh et al. (2023) demonstrated that an $\alpha$-SOSP can be found with $\alpha=O(\frac{1}{n^{1/3}}+(\frac{\sqrt{d}}{n\epsilon})^{3/7})$, where $n$ is the dataset size, $d$ is the dimension, and $\epsilon$ is the differential privacy parameter. Building on the SpiderBoost algorithm framework, we propose a new approach that uses adaptive batch sizes and incorporates the binary tree mechanism. Our method improves the results for privately finding an SOSP, achieving $\alpha=O(\frac{1}{n^{1/3}}+(\frac{\sqrt{d}}{n\epsilon})^{1/2})$. This improved bound matches the state-of-the-art for finding an FOSP, suggesting that privately finding an SOSP may be achievable at no additional cost.

Title: Using LLMs to Discover Legal Factors

• Authors: Morgan Gray, Jaromir Savelka, Wesley Oliver, Kevin Ashley
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07504
• Pdf URL: https://arxiv.org/pdf/2410.07504
• Copy Paste: [[2410.07504]] Using LLMs to Discover Legal Factors(https://arxiv.org/abs/2410.07504)
• Keywords: large language model
• Abstract: Factors are a foundational component of legal analysis and computational models of legal reasoning. These factor-based representations enable lawyers, judges, and AI and Law researchers to reason about legal cases. In this paper, we introduce a methodology that leverages large language models (LLMs) to discover lists of factors that effectively represent a legal domain. Our method takes as input raw court opinions and produces a set of factors and associated definitions. We demonstrate that a semi-automated approach, incorporating minimal human involvement, produces factor representations that can predict case outcomes with moderate success, if not yet as well as expert-defined factors can.

Title: CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression

• Authors: Wenyuan Liu, Xindian Ma, Peng Zhang, Yan Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07505
• Pdf URL: https://arxiv.org/pdf/2410.07505
• Copy Paste: [[2410.07505]] CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression(https://arxiv.org/abs/2410.07505)
• Keywords: large language model
• Abstract: Post-Training Quantization (PTQ) is an effective technique for compressing Large Language Models (LLMs). While many studies focus on quantizing both weights and activations, it is still a challenge to maintain the accuracy of LLM after activating quantization. To investigate the primary cause, we extend the concept of kernel from linear algebra to quantization functions to define a new term, "quantization kernel", which refers to the set of elements in activations that are quantized to zero. Through quantitative analysis of the quantization kernel, we find that these elements are crucial for maintaining the accuracy of quantized LLMs. With the decrease of quantization kernel, the precision of quantized LLMs increases. If the quantization kernel proportion is kept below 19% for OPT models and below 1% for LLaMA models, the precision loss from quantizing activations to INT8 becomes negligible. Motivated by the goal of developing a quantization method with small quantization kernel, we propose CrossQuant: a simple yet effective method for quantizing activations. CrossQuant cross-quantizes elements using row and column-wise absolute maximum vectors, achieving a quantization kernel of approximately 16% for OPT models and less than 0.1% for LLaMA models. Experimental results on LLMs (LLaMA, OPT) ranging from 6.7B to 70B parameters demonstrate that CrossQuant improves or maintains perplexity and accuracy in language modeling, zero-shot, and few-shot tasks.

Title: Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)

• Authors: Abhijit Mishra, Shreya Shukla, Jose Torres, Jacek Gwizdka, Shounak Roychowdhury
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07507
• Pdf URL: https://arxiv.org/pdf/2410.07507
• Copy Paste: [[2410.07507]] Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)(https://arxiv.org/abs/2410.07507)
• Keywords: extraction, large language model
• Abstract: Decoding and expressing brain activity in a comprehensible form is a challenging frontier in AI. This paper presents Thought2Text, which uses instruction-tuned Large Language Models (LLMs) fine-tuned with EEG data to achieve this goal. The approach involves three stages: (1) training an EEG encoder for visual feature extraction, (2) fine-tuning LLMs on image and text data, enabling multimodal description generation, and (3) further fine-tuning on EEG embeddings to generate text directly from EEG during inference. Experiments on a public EEG dataset collected for six subjects with image stimuli demonstrate the efficacy of multimodal LLMs (LLaMa-v3, Mistral-v0.3, Qwen2.5), validated using traditional language generation evaluation metrics, GPT-4 based assessments, and evaluations by human expert. This approach marks a significant advancement towards portable, low-cost "thoughts-to-text" technology with potential applications in both neuroscience and natural language processing (NLP).

Title: CSGDN: Contrastive Signed Graph Diffusion Network for Predicting Crop Gene-Trait Associations

• Authors: Yiru Pan, Xingyu Ji, Jiaqi You, Lu Li, Zhenping Liu, Xianlong Zhang, Zeyu Zhang, Maojun Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07511
• Pdf URL: https://arxiv.org/pdf/2410.07511
• Copy Paste: [[2410.07511]] CSGDN: Contrastive Signed Graph Diffusion Network for Predicting Crop Gene-Trait Associations(https://arxiv.org/abs/2410.07511)
• Keywords: robust, diffusion
• Abstract: Positive and negative association preidiction between gene and trait help studies for crops to perform complex physiological functions. The transcription and regulation activity of specific genes will be adjusted accordingly in different cell types, developmental stages, and physiological states to meet the needs of organisms. Determing gene-trait associations can resolve the mechanism of trait formation and benefit the improvement of crop yield and quality. There are the following two problems in obtaining the positive/negative associations between gene and trait: 1) High-throughput DNA/RNA sequencing and trait data collection are expensive and time-consuming due to the need to process large sample sizes; 2) experiments introduce both random and systematic errors, and, at the same time, calculations or predictions using software or models may produce noise. To address these two issues, we propose a Contrastive Signed Graph Diffusion Network, CSGDN, to learn robust node representations with fewer training samples to achieve higher link prediction accuracy. CSGDN employs a signed graph diffusion method to uncover the underlying regulatory associations between genes and traits. Then, stochastic perterbation strategies are used to create two views for both original and diffusive graphs. At last, a multi-view contrastive learning paradigm loss is designed to unify the node presentations learned from the two views to resist interference and reduce noise. We conduct experiments to validate the performance of CSGDN on three crop datasets: Gossypium hirsutum, Brassica napus, and Triticum turgidum. The results demonstrate that the proposed model outperforms state-of-the-art methods by up to 9.28% AUC for link sign prediction in G. hirsutum dataset.

Title: Evolutionary Contrastive Distillation for Language Model Alignment

• Authors: Julian Katz-Samuels, Zheng Li, Hyokun Yun, Priyanka Nigam, Yi Xu, Vaclav Petricek, Bing Yin, Trishul Chilimbi
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07513
• Pdf URL: https://arxiv.org/pdf/2410.07513
• Copy Paste: [[2410.07513]] Evolutionary Contrastive Distillation for Language Model Alignment(https://arxiv.org/abs/2410.07513)
• Keywords: large language model
• Abstract: The ability of large language models (LLMs) to execute complex instructions is essential for their real-world applications. However, several recent studies indicate that LLMs struggle with challenging instructions. In this paper, we propose Evolutionary Contrastive Distillation (ECD), a novel method for generating high-quality synthetic preference data designed to enhance the complex instruction-following capability of language models. ECD generates data that specifically illustrates the difference between a response that successfully follows a set of complex instructions and a response that is high-quality, but nevertheless makes some subtle mistakes. This is done by prompting LLMs to progressively evolve simple instructions to more complex instructions. When the complexity of an instruction is increased, the original successful response to the original instruction becomes a "hard negative" response for the new instruction, mostly meeting requirements of the new instruction, but barely missing one or two. By pairing a good response with such a hard negative response, and employing contrastive learning algorithms such as DPO, we improve language models' ability to follow complex instructions. Empirically, we observe that our method yields a 7B model that exceeds the complex instruction-following performance of current SOTA 7B models and is competitive even with open-source 70B models.

Title: News Reporter: A Multi-lingual LLM Framework for Broadcast T.V News

• Authors: Tarun Jain, Yufei Gao, Sridhar Vanga, Karan Singla
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07520
• Pdf URL: https://arxiv.org/pdf/2410.07520
• Copy Paste: [[2410.07520]] News Reporter: A Multi-lingual LLM Framework for Broadcast T.V News(https://arxiv.org/abs/2410.07520)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have fast become an essential tools to many conversational chatbots due to their ability to provide coherent answers for varied queries. Datasets used to train these LLMs are often a mix of generic and synthetic samples, thus lacking the verification needed to provide correct and verifiable answers for T.V. News. We collect and share a large collection of QA pairs extracted from transcripts of news recordings from various news-channels across the United States. Resultant QA pairs are then used to fine-tune an off-the-shelf LLM model. Our model surpasses base models of similar size on several open LLM benchmarks. We further integrate and propose a RAG method to improve contextualization of our answers and also point it to a verifiable news recording.

Title: DemoShapley: Valuation of Demonstrations for In-Context Learning

• Authors: Shan Xie, Man Luo, Chadly Daniel Stern, Mengnan Du, Lu Cheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07523
• Pdf URL: https://arxiv.org/pdf/2410.07523
• Copy Paste: [[2410.07523]] DemoShapley: Valuation of Demonstrations for In-Context Learning(https://arxiv.org/abs/2410.07523)
• Keywords: fair, large language model
• Abstract: Large language models (LLMs) leveraging in-context learning (ICL) have set new benchmarks in few-shot learning across various tasks without needing task-specific fine-tuning. However, extensive research has demonstrated that the effectiveness of ICL is significantly influenced by the selection and ordering of demonstrations. Considering the critical role of demonstration selection in ICL, we introduce DemoShapley which is inspired by the Data Shapley valuation theorem. This approach assesses the influence of individual demonstration instances, distinguishing between those that contribute positively and those that may hinder performance. Our findings reveal that DemoShapley not only enhances model performance in terms of accuracy and fairness but also generalizes queries from domains distinct from those of the in-context demonstrations, highlighting its versatility and effectiveness in optimizing ICL demonstration selection. Last but not least, DemoShapley demonstrates its ability to aid in identifying noisy data within the demonstration set.

Title: Upcycling Large Language Models into Mixture of Experts

• Authors: Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, Bryan Catanzaro
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07524
• Pdf URL: https://arxiv.org/pdf/2410.07524
• Copy Paste: [[2410.07524]] Upcycling Large Language Models into Mixture of Experts(https://arxiv.org/abs/2410.07524)
• Keywords: large language model
• Abstract: Upcycling pre-trained dense language models into sparse mixture-of-experts (MoE) models is an efficient approach to increase the model capacity of already trained models. However, optimal techniques for upcycling at scale remain unclear. In this work, we conduct an extensive study of upcycling methods and hyperparameters for billion-parameter scale language models. We propose a novel "virtual group" initialization scheme and weight scaling approach to enable upcycling into fine-grained MoE architectures. Through ablations, we find that upcycling outperforms continued dense model training. In addition, we show that softmax-then-topK expert routing improves over topK-then-softmax approach and higher granularity MoEs can help improve accuracy. Finally, we upcycled Nemotron-4 15B on 1T tokens and compared it to a continuously trained version of the same model on the same 1T tokens: the continuous trained model achieved 65.3% MMLU, whereas the upcycled model achieved 67.6%. Our results offer insights and best practices to effectively leverage upcycling for building MoE language models.

Title: Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare

• Authors: Nan Fang, Guiliang Liu, Wei Gong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07525
• Pdf URL: https://arxiv.org/pdf/2410.07525
• Copy Paste: [[2410.07525]] Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare(https://arxiv.org/abs/2410.07525)
• Keywords: transformer, generative
• Abstract: Reinforcement Learning (RL) applied in healthcare can lead to unsafe medical decisions and treatment, such as excessive dosages or abrupt changes, often due to agents overlooking common-sense constraints. Consequently, Constrained Reinforcement Learning (CRL) is a natural choice for safe decisions. However, specifying the exact cost function is inherently difficult in healthcare. Recent Inverse Constrained Reinforcement Learning (ICRL) is a promising approach that infers constraints from expert demonstrations. ICRL algorithms model Markovian decisions in an interactive environment. These settings do not align with the practical requirement of a decision-making system in healthcare, where decisions rely on historical treatment recorded in an offline dataset. To tackle these issues, we propose the Constraint Transformer (CT). Specifically, 1) we utilize a causal attention mechanism to incorporate historical decisions and observations into the constraint modeling, while employing a Non-Markovian layer for weighted constraints to capture critical states. 2) A generative world model is used to perform exploratory data augmentation, enabling offline RL methods to simulate unsafe decision sequences. In multiple medical scenarios, empirical results demonstrate that CT can capture unsafe states and achieve strategies that approximate lower mortality rates, reducing the occurrence probability of unsafe behaviors.

Title: MKGL: Mastery of a Three-Word Language

• Authors: Lingbing Guo, Zhongpu Bo, Zhuo Chen, Yichi Zhang, Jiaoyan Chen, Yarong Lan, Mengshu Sun, Zhiqiang Zhang, Yangyifei Luo, Qian Li, Qiang Zhang, Wen Zhang, Huajun Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07526
• Pdf URL: https://arxiv.org/pdf/2410.07526
• Copy Paste: [[2410.07526]] MKGL: Mastery of a Three-Word Language(https://arxiv.org/abs/2410.07526)
• Keywords: large language model
• Abstract: Large language models (LLMs) have significantly advanced performance across a spectrum of natural language processing (NLP) tasks. Yet, their application to knowledge graphs (KGs), which describe facts in the form of triplets and allow minimal hallucinations, remains an underexplored frontier. In this paper, we investigate the integration of LLMs with KGs by introducing a specialized KG Language (KGL), where a sentence precisely consists of an entity noun, a relation verb, and ends with another entity noun. Despite KGL's unfamiliar vocabulary to the LLM, we facilitate its learning through a tailored dictionary and illustrative sentences, and enhance context understanding via real-time KG context retrieval and KGL token embedding augmentation. Our results reveal that LLMs can achieve fluency in KGL, drastically reducing errors compared to conventional KG embedding methods on KG completion. Furthermore, our enhanced LLM shows exceptional competence in generating accurate three-word sentences from an initial entity and interpreting new unseen terms out of KGs.

Title: Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification

• Authors: Haolin Liu, Artin Tajdini, Andrew Wagenmaker, Chen-Yu Wei
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07533
• Pdf URL: https://arxiv.org/pdf/2410.07533
• Copy Paste: [[2410.07533]] Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification(https://arxiv.org/abs/2410.07533)
• Keywords: robust
• Abstract: In linear bandits, how can a learner effectively learn when facing corrupted rewards? While significant work has explored this question, a holistic understanding across different adversarial models and corruption measures is lacking, as is a full characterization of the minimax regret bounds. In this work, we compare two types of corruptions commonly considered: strong corruption, where the corruption level depends on the action chosen by the learner, and weak corruption, where the corruption level does not depend on the action chosen by the learner. We provide a unified framework to analyze these corruptions. For stochastic linear bandits, we fully characterize the gap between the minimax regret under strong and weak corruptions. We also initiate the study of corrupted adversarial linear bandits, obtaining upper and lower bounds with matching dependencies on the corruption level. Next, we reveal a connection between corruption-robust learning and learning with gap-dependent mis-specification, a setting first studied by Liu et al. (2023a), where the misspecification level of an action or policy is proportional to its suboptimality. We present a general reduction that enables any corruption-robust algorithm to handle gap-dependent misspecification. This allows us to recover the results of Liu et al. (2023a) in a black-box manner and significantly generalize them to settings like linear MDPs, yielding the first results for gap-dependent misspecification in reinforcement learning. However, this general reduction does not attain the optimal rate for gap-dependent misspecification. Motivated by this, we develop a specialized algorithm that achieves optimal bounds for gap-dependent misspecification in linear bandits, thus answering an open question posed by Liu et al. (2023a).

Title: I-Max: Maximize the Resolution Potential of Pre-trained Rectified Flow Transformers with Projected Flow

• Authors: Ruoyi Du, Dongyang Liu, Le Zhuo, Qin Qi, Hongsheng Li, Zhanyu Ma, Peng Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07536
• Pdf URL: https://arxiv.org/pdf/2410.07536
• Copy Paste: [[2410.07536]] I-Max: Maximize the Resolution Potential of Pre-trained Rectified Flow Transformers with Projected Flow(https://arxiv.org/abs/2410.07536)
• Keywords: diffusion, transformer, generative
• Abstract: Rectified Flow Transformers (RFTs) offer superior training and inference efficiency, making them likely the most viable direction for scaling up diffusion models. However, progress in generation resolution has been relatively slow due to data quality and training costs. Tuning-free resolution extrapolation presents an alternative, but current methods often reduce generative stability, limiting practical application. In this paper, we review existing resolution extrapolation methods and introduce the I-Max framework to maximize the resolution potential of Text-to-Image RFTs. I-Max features: (i) a novel Projected Flow strategy for stable extrapolation and (ii) an advanced inference toolkit for generalizing model knowledge to higher resolutions. Experiments with Lumina-Next-2K and Flux.1-dev demonstrate I-Max's ability to enhance stability in resolution extrapolation and show that it can bring image detail emergence and artifact correction, confirming the practical value of tuning-free resolution extrapolation.

Title: OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting

• Authors: Xukai Liu, Ye Liu, Kai Zhang, Kehang Wang, Qi Liu, Enhong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07549
• Pdf URL: https://arxiv.org/pdf/2410.07549
• Copy Paste: [[2410.07549]] OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting(https://arxiv.org/abs/2410.07549)
• Keywords: large language model
• Abstract: Entity Linking (EL) is the process of associating ambiguous textual mentions to specific entities in a knowledge base. Traditional EL methods heavily rely on large datasets to enhance their performance, a dependency that becomes problematic in the context of few-shot entity linking, where only a limited number of examples are available for training. To address this challenge, we present OneNet, an innovative framework that utilizes the few-shot learning capabilities of Large Language Models (LLMs) without the need for fine-tuning. To the best of our knowledge, this marks a pioneering approach to applying LLMs to few-shot entity linking tasks. OneNet is structured around three key components prompted by LLMs: (1) an entity reduction processor that simplifies inputs by summarizing and filtering out irrelevant entities, (2) a dual-perspective entity linker that combines contextual cues and prior knowledge for precise entity linking, and (3) an entity consensus judger that employs a unique consistency algorithm to alleviate the hallucination in the entity linking reasoning. Comprehensive evaluations across seven benchmark datasets reveal that OneNet outperforms current state-of-the-art entity linking methods.

Title: Conditional Lagrangian Wasserstein Flow for Time Series Imputation

• Authors: Weizhu Qian, Dalin Zhang, Yan Zhao
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07550
• Pdf URL: https://arxiv.org/pdf/2410.07550
• Copy Paste: [[2410.07550]] Conditional Lagrangian Wasserstein Flow for Time Series Imputation(https://arxiv.org/abs/2410.07550)
• Keywords: diffusion
• Abstract: Time series imputation is important for numerous real-world applications. To overcome the limitations of diffusion model-based imputation methods, e.g., slow convergence in inference, we propose a novel method for time series imputation in this work, called Conditional Lagrangian Wasserstein Flow. The proposed method leverages the (conditional) optimal transport theory to learn the probability flow in a simulation-free manner, in which the initial noise, missing data, and observations are treated as the source distribution, target distribution, and conditional information, respectively. According to the principle of least action in Lagrangian mechanics, we learn the velocity by minimizing the corresponding kinetic energy. Moreover, to incorporate more prior information into the model, we parameterize the derivative of a task-specific potential function via a variational autoencoder, and combine it with the base estimator to formulate a Rao-Blackwellized sampler. The propose model allows us to take less intermediate steps to produce high-quality samples for inference compared to existing diffusion methods. Finally, the experimental results on the real-word datasets show that the proposed method achieves competitive performance on time series imputation compared to the state-of-the-art methods.

Title: KRAG Framework for Enhancing LLMs in the Legal Domain

• Authors: Nguyen Ha Thanh, Ken Satoh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07551
• Pdf URL: https://arxiv.org/pdf/2410.07551
• Copy Paste: [[2410.07551]] KRAG Framework for Enhancing LLMs in the Legal Domain(https://arxiv.org/abs/2410.07551)
• Keywords: large language model
• Abstract: This paper introduces Knowledge Representation Augmented Generation (KRAG), a novel framework designed to enhance the capabilities of Large Language Models (LLMs) within domain-specific applications. KRAG points to the strategic inclusion of critical knowledge entities and relationships that are typically absent in standard data sets and which LLMs do not inherently learn. In the context of legal applications, we present Soft PROLEG, an implementation model under KRAG, which uses inference graphs to aid LLMs in delivering structured legal reasoning, argumentation, and explanations tailored to user inquiries. The integration of KRAG, either as a standalone framework or in tandem with retrieval augmented generation (RAG), markedly improves the ability of language models to navigate and solve the intricate challenges posed by legal texts and terminologies. This paper details KRAG's methodology, its implementation through Soft PROLEG, and potential broader applications, underscoring its significant role in advancing natural language understanding and processing in specialized knowledge domains.

Title: AI-Press: A Multi-Agent News Generating and Feedback Simulation System Powered by Large Language Models

• Authors: Xiawei Liu, Shiyue Yang, Xinnong Zhang, Haoyu Kuang, Libo Sun, Yihang Yang, Siming Chen, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07561
• Pdf URL: https://arxiv.org/pdf/2410.07561
• Copy Paste: [[2410.07561]] AI-Press: A Multi-Agent News Generating and Feedback Simulation System Powered by Large Language Models(https://arxiv.org/abs/2410.07561)
• Keywords: large language model
• Abstract: The rise of various social platforms has transformed journalism. The growing demand for news content has led to the increased use of large language models (LLMs) in news production due to their speed and cost-effectiveness. However, LLMs still encounter limitations in professionalism and ethical judgment in news generation. Additionally, predicting public feedback is usually difficult before news is released. To tackle these challenges, we introduce AI-Press, an automated news drafting and polishing system based on multi-agent collaboration and Retrieval-Augmented Generation. We develop a feedback simulation system that generates public feedback considering demographic distributions. Through extensive quantitative and qualitative evaluations, our system shows significant improvements in news-generating capabilities and verifies the effectiveness of public feedback simulation.

Title: When and Where Did it Happen? An Encoder-Decoder Model to Identify Scenario Context

• Authors: Enrique Noriega-Atala, Robert Vacareanu, Salena Torres Ashton, Adarsh Pyarelal, Clayton T. Morrison, Mihai Surdeanu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07567
• Pdf URL: https://arxiv.org/pdf/2410.07567
• Copy Paste: [[2410.07567]] When and Where Did it Happen? An Encoder-Decoder Model to Identify Scenario Context(https://arxiv.org/abs/2410.07567)
• Keywords: extraction
• Abstract: We introduce a neural architecture finetuned for the task of scenario context generation: The relevant location and time of an event or entity mentioned in text. Contextualizing information extraction helps to scope the validity of automated finings when aggregating them as knowledge graphs. Our approach uses a high-quality curated dataset of time and location annotations in a corpus of epidemiology papers to train an encoder-decoder architecture. We also explored the use of data augmentation techniques during training. Our findings suggest that a relatively small fine-tuned encoder-decoder model performs better than out-of-the-box LLMs and semantic role labeling parsers to accurate predict the relevant scenario information of a particular entity or event.

Title: How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?

• Authors: Seongyun Lee, Geewook Kim, Jiyeon Kim, Hyunji Lee, Hoyeon Chang, Sue Hyun Park, Minjoon Seo
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07571
• Pdf URL: https://arxiv.org/pdf/2410.07571
• Copy Paste: [[2410.07571]] How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?(https://arxiv.org/abs/2410.07571)
• Keywords: secure, large language model
• Abstract: Vision-Language adaptation (VL adaptation) transforms Large Language Models (LLMs) into Large Vision-Language Models (LVLMs) for multimodal tasks, but this process often compromises the inherent safety capabilities embedded in the original LLMs. Despite potential harmfulness due to weakened safety measures, in-depth analysis on the effects of VL adaptation on safety remains under-explored. This study examines how VL adaptation influences safety and evaluates the impact of safety fine-tuning methods. Our analysis reveals that safety degradation occurs during VL adaptation, even when the training data is safe. While safety tuning techniques like supervised fine-tuning with safety datasets or reinforcement learning from human feedback mitigate some risks, they still lead to safety degradation and a reduction in helpfulness due to over-rejection issues. Further analysis of internal model weights suggests that VL adaptation may impact certain safety-related layers, potentially lowering overall safety levels. Additionally, our findings demonstrate that the objectives of VL adaptation and safety tuning are divergent, which often results in their simultaneous application being suboptimal. To address this, we suggest the weight merging approach as an optimal solution effectively reducing safety degradation while maintaining helpfulness. These insights help guide the development of more reliable and secure LVLMs for real-world applications.

Title: RealVul: Can We Detect Vulnerabilities in Web Applications with LLM?

• Authors: Di Cao, Yong Liao, Xiuwei Shang
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07573
• Pdf URL: https://arxiv.org/pdf/2410.07573
• Copy Paste: [[2410.07573]] RealVul: Can We Detect Vulnerabilities in Web Applications with LLM?(https://arxiv.org/abs/2410.07573)
• Keywords: large language model
• Abstract: The latest advancements in large language models (LLMs) have sparked interest in their potential for software vulnerability detection. However, there is currently a lack of research specifically focused on vulnerabilities in the PHP language, and challenges in extracting samples and processing persist, hindering the model's ability to effectively capture the characteristics of specific vulnerabilities. In this paper, we present RealVul, the first LLM-based framework designed for PHP vulnerability detection, addressing these issues. By vulnerability candidate detection methods and employing techniques such as normalization, we can isolate potential vulnerability triggers while streamlining the code and eliminating unnecessary semantic information, enabling the model to better understand and learn from the generated vulnerability samples. We also address the issue of insufficient PHP vulnerability samples by improving data synthesis methods. To evaluate RealVul's performance, we conduct an extensive analysis using five distinct code LLMs on vulnerability data from 180 PHP projects. The results demonstrate a significant improvement in both effectiveness and generalization compared to existing methods, effectively boosting the vulnerability detection capabilities of these models.

Title: 3D Vision-Language Gaussian Splatting

• Authors: Qucheng Peng, Benjamin Planche, Zhongpai Gao, Meng Zheng, Anwesa Choudhuri, Terrence Chen, Chen Chen, Ziyan Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07577
• Pdf URL: https://arxiv.org/pdf/2410.07577
• Copy Paste: [[2410.07577]] 3D Vision-Language Gaussian Splatting(https://arxiv.org/abs/2410.07577)
• Keywords: segmentation
• Abstract: Recent advancements in 3D reconstruction methods and vision-language models have propelled the development of multi-modal 3D scene understanding, which has vital applications in robotics, autonomous driving, and virtual/augmented reality. However, current multi-modal scene understanding approaches have naively embedded semantic representations into 3D reconstruction methods without striking a balance between visual and language modalities, which leads to unsatisfying semantic rasterization of translucent or reflective objects, as well as over-fitting on color modality. To alleviate these limitations, we propose a solution that adequately handles the distinct visual and semantic modalities, i.e., a 3D vision-language Gaussian splatting model for scene understanding, to put emphasis on the representation learning of language modality. We propose a novel cross-modal rasterizer, using modality fusion along with a smoothed semantic indicator for enhancing semantic rasterization. We also employ a camera-view blending technique to improve semantic consistency between existing and synthesized views, thereby effectively mitigating over-fitting. Extensive experiments demonstrate that our method achieves state-of-the-art performance in open-vocabulary semantic segmentation, surpassing existing methods by a significant margin.

Title: Detecting Training Data of Large Language Models via Expectation Maximization

• Authors: Gyuwan Kim, Yang Li, Evangelia Spiliopoulou, Jie Ma, Miguel Ballesteros, William Yang Wang
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07582
• Pdf URL: https://arxiv.org/pdf/2410.07582
• Copy Paste: [[2410.07582]] Detecting Training Data of Large Language Models via Expectation Maximization(https://arxiv.org/abs/2410.07582)
• Keywords: privacy, attack, robust, membership infer, large language model
• Abstract: The widespread deployment of large language models (LLMs) has led to impressive advancements, yet information about their training data, a critical factor in their performance, remains undisclosed. Membership inference attacks (MIAs) aim to determine whether a specific instance was part of a target model's training data. MIAs can offer insights into LLM outputs and help detect and address concerns such as data contamination and compliance with privacy and copyright standards. However, applying MIAs to LLMs presents unique challenges due to the massive scale of pre-training data and the ambiguous nature of membership. Additionally, creating appropriate benchmarks to evaluate MIA methods is not straightforward, as training and test data distributions are often unknown. In this paper, we introduce EM-MIA, a novel MIA method for LLMs that iteratively refines membership scores and prefix scores via an expectation-maximization algorithm, leveraging the duality that the estimates of these scores can be improved by each other. Membership scores and prefix scores assess how each instance is likely to be a member and discriminative as a prefix, respectively. Our method achieves state-of-the-art results on the WikiMIA dataset. To further evaluate EM-MIA, we present OLMoMIA, a benchmark built from OLMo resources, which allows us to control the difficulty of MIA tasks with varying degrees of overlap between training and test data distributions. We believe that EM-MIA serves as a robust MIA method for LLMs and that OLMoMIA provides a valuable resource for comprehensively evaluating MIA approaches, thereby driving future research in this critical area.

Title: TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text

• Authors: Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, Yaohua Tang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07590
• Pdf URL: https://arxiv.org/pdf/2410.07590
• Copy Paste: [[2410.07590]] TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text(https://arxiv.org/abs/2410.07590)
• Keywords: large language model
• Abstract: Current Retrieval-Augmented Generation (RAG) systems concatenate and process numerous retrieved document chunks for prefill which requires a large volume of computation, therefore leading to significant latency in time-to-first-token (TTFT). To reduce the computation overhead as well as TTFT, we introduce TurboRAG, a novel RAG system that redesigns the inference paradigm of the current RAG system by first pre-computing and storing the key-value (KV) caches of documents offline, and then directly retrieving the saved KV cache for prefill. Hence, online computation of KV caches is eliminated during inference. In addition, we provide a number of insights into the mask matrix and positional embedding mechanisms, plus fine-tune a pretrained language model to maintain model accuracy of TurboRAG. Our approach is applicable to most existing large language models and their applications without any requirement in modification of models and inference systems. Experimental results across a suite of RAG benchmarks demonstrate that TurboRAG reduces TTFT by up to 9.4x compared to the conventional RAG systems (on an average of 8.6x), but reserving comparable performance to the standard RAG systems.

Title: A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks

• Authors: Hoin Jung, Taeuk Jang, Xiaoqian Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07593
• Pdf URL: https://arxiv.org/pdf/2410.07593
• Copy Paste: [[2410.07593]] A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks(https://arxiv.org/abs/2410.07593)
• Keywords: fair
• Abstract: Recent advancements in Vision-Language Models (VLMs) have enabled complex multimodal tasks by processing text and image data simultaneously, significantly enhancing the field of artificial intelligence. However, these models often exhibit biases that can skew outputs towards societal stereotypes, thus necessitating debiasing strategies. Existing debiasing methods focus narrowly on specific modalities or tasks, and require extensive retraining. To address these limitations, this paper introduces Selective Feature Imputation for Debiasing (SFID), a novel methodology that integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs. SFID is versatile, maintaining the semantic integrity of outputs and costly effective by eliminating the need for retraining. Our experimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without compromising performance. This approach not only enhances the fairness of VLMs applications but also preserves their efficiency and utility across diverse scenarios.

Title: Fine-detailed Neural Indoor Scene Reconstruction using multi-level importance sampling and multi-view consistency

• Authors: Xinghui Li, Yuchen Ji, Xiansong Lai, Wanting Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07597
• Pdf URL: https://arxiv.org/pdf/2410.07597
• Copy Paste: [[2410.07597]] Fine-detailed Neural Indoor Scene Reconstruction using multi-level importance sampling and multi-view consistency(https://arxiv.org/abs/2410.07597)
• Keywords: segmentation
• Abstract: Recently, neural implicit 3D reconstruction in indoor scenarios has become popular due to its simplicity and impressive performance. Previous works could produce complete results leveraging monocular priors of normal or depth. However, they may suffer from over-smoothed reconstructions and long-time optimization due to unbiased sampling and inaccurate monocular priors. In this paper, we propose a novel neural implicit surface reconstruction method, named FD-NeuS, to learn fine-detailed 3D models using multi-level importance sampling strategy and multi-view consistency methodology. Specifically, we leverage segmentation priors to guide region-based ray sampling, and use piecewise exponential functions as weights to pilot 3D points sampling along the rays, ensuring more attention on important regions. In addition, we introduce multi-view feature consistency and multi-view normal consistency as supervision and uncertainty respectively, which further improve the reconstruction of details. Extensive quantitative and qualitative results show that FD-NeuS outperforms existing methods in various scenes.

Title: Causal Image Modeling for Efficient Visual Understanding

• Authors: Feng Wang, Timing Yang, Yaodong Yu, Sucheng Ren, Guoyizhe Wei, Angtian Wang, Wei Shao, Yuyin Zhou, Alan Yuille, Cihang Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07599
• Pdf URL: https://arxiv.org/pdf/2410.07599
• Copy Paste: [[2410.07599]] Causal Image Modeling for Efficient Visual Understanding(https://arxiv.org/abs/2410.07599)
• Keywords: transformer
• Abstract: In this work, we present a comprehensive analysis of causal image modeling and introduce the Adventurer series models where we treat images as sequences of patch tokens and employ uni-directional language models to learn visual representations. This modeling paradigm allows us to process images in a recurrent formulation with linear complexity relative to the sequence length, which can effectively address the memory and computation explosion issues posed by high-resolution and fine-grained images. In detail, we introduce two simple designs that seamlessly integrate image inputs into the causal inference framework: a global pooling token placed at the beginning of the sequence and a flipping operation between every two layers. Extensive empirical studies demonstrate the significant efficiency and effectiveness of this causal image modeling paradigm. For example, our base-sized Adventurer model attains a competitive test accuracy of 84.0% on the standard ImageNet-1k benchmark with 216 images/s training throughput, which is 5.3 times more efficient than vision transformers to achieve the same result.

Title: RNA: Video Editing with ROI-based Neural Atlas

• Authors: Jaekyeong Lee, Geonung Kim, Sunghyun Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07600
• Pdf URL: https://arxiv.org/pdf/2410.07600
• Copy Paste: [[2410.07600]] RNA: Video Editing with ROI-based Neural Atlas(https://arxiv.org/abs/2410.07600)
• Keywords: segmentation
• Abstract: With the recent growth of video-based Social Network Service (SNS) platforms, the demand for video editing among common users has increased. However, video editing can be challenging due to the temporally-varying factors such as camera movement and moving objects. While modern atlas-based video editing methods have addressed these issues, they often fail to edit videos including complex motion or multiple moving objects, and demand excessive computational cost, even for very simple edits. In this paper, we propose a novel region-of-interest (ROI)-based video editing framework: ROI-based Neural Atlas (RNA). Unlike prior work, RNA allows users to specify editing regions, simplifying the editing process by removing the need for foreground separation and atlas modeling for foreground objects. However, this simplification presents a unique challenge: acquiring a mask that effectively handles occlusions in the edited area caused by moving objects, without relying on an additional segmentation model. To tackle this, we propose a novel mask refinement approach designed for this specific challenge. Moreover, we introduce a soft neural atlas model for video reconstruction to ensure high-quality editing results. Extensive experiments show that RNA offers a more practical and efficient editing solution, applicable to a wider range of videos with superior quality compared to prior methods.

Title: A Variational Bayesian Inference Theory of Elasticity and Its Mixed Probabilistic Finite Element Method for Inverse Deformation Solutions in Any Dimension

• Authors: Chao Wang, Shaofan Li
• Subjects: cs.CV, cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2410.07605
• Pdf URL: https://arxiv.org/pdf/2410.07605
• Copy Paste: [[2410.07605]] A Variational Bayesian Inference Theory of Elasticity and Its Mixed Probabilistic Finite Element Method for Inverse Deformation Solutions in Any Dimension(https://arxiv.org/abs/2410.07605)
• Keywords: robust
• Abstract: In this work, we have developed a variational Bayesian inference theory of elasticity, which is accomplished by using a mixed Variational Bayesian inference Finite Element Method (VBI-FEM) that can be used to solve the inverse deformation problems of continua. In the proposed variational Bayesian inference theory of continuum mechanics, the elastic strain energy is used as a prior in a Bayesian inference network, which can intelligently recover the detailed continuum deformation mappings with only given the information on the deformed and undeformed continuum body shapes without knowing the interior deformation and the precise actual boundary conditions, both traction as well as displacement boundary conditions, and the actual material constitutive relation. Moreover, we have implemented the related finite element formulation in a computational probabilistic mechanics framework. To numerically solve mixed variational problem, we developed an operator splitting or staggered algorithm that consists of the finite element (FE) step and the Bayesian learning (BL) step as an analogue of the well-known the Expectation-Maximization (EM) algorithm. By solving the mixed probabilistic Galerkin variational problem, we demonstrated that the proposed method is able to inversely predict continuum deformation mappings with strong discontinuity or fracture without knowing the external load conditions. The proposed method provides a robust machine intelligent solution for the long-sought-after inverse problem solution, which has been a major challenge in structure failure forensic pattern analysis in past several decades. The proposed method may become a promising artificial intelligence-based inverse method for solving general partial differential equations.

Title: Parallel Digital Twin-driven Deep Reinforcement Learning for User Association and Load Balancing in Dynamic Wireless Networks

• Authors: Zhenyu Tao, Wei Xu, Xiaohu You
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.07611
• Pdf URL: https://arxiv.org/pdf/2410.07611
• Copy Paste: [[2410.07611]] Parallel Digital Twin-driven Deep Reinforcement Learning for User Association and Load Balancing in Dynamic Wireless Networks(https://arxiv.org/abs/2410.07611)
• Keywords: diffusion, generative
• Abstract: Optimization of user association in a densely deployed heterogeneous cellular network is usually challenging and even more complicated due to the dynamic nature of user mobility and fluctuation in user counts. While deep reinforcement learning (DRL) emerges as a promising solution, its application in practice is hindered by high trial-and-error costs in real world and unsatisfactory physical network performance during training. In addition, existing DRL-based user association methods are usually only applicable to scenarios with a fixed number of users due to convergence and compatibility challenges. In this paper, we propose a parallel digital twin (DT)-driven DRL method for user association and load balancing in networks with both dynamic user counts, distribution, and mobility patterns. Our method employs a distributed DRL strategy to handle varying user numbers and exploits a refined neural network structure for faster convergence. To address these DRL training-related challenges, we devise a high-fidelity DT construction technique, featuring a zero-shot generative user mobility model, named Map2Traj, based on a diffusion model. Map2Traj estimates user trajectory patterns and spatial distributions solely from street maps. Armed with this DT environment, DRL agents are enabled to be trained without the need for interactions with the physical network. To enhance the generalization ability of DRL models for dynamic scenarios, a parallel DT framework is further established to alleviate strong correlation and non-stationarity in single-environment training and improve the training efficiency. Numerical results show that the proposed parallel DT-driven DRL method achieves closely comparable performance to real environment training, and even outperforms those trained in a single real-world environment with nearly 20% gain in terms of cell-edge user performance.

Title: A Survey for Deep Reinforcement Learning Based Network Intrusion Detection

• Authors: Wanrong Yang, Alberto Acuto, Yihang Zhou, Dominik Wojtczak
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07612
• Pdf URL: https://arxiv.org/pdf/2410.07612
• Copy Paste: [[2410.07612]] A Survey for Deep Reinforcement Learning Based Network Intrusion Detection(https://arxiv.org/abs/2410.07612)
• Keywords: attack, robust, generative
• Abstract: Cyber-attacks are becoming increasingly sophisticated and frequent, highlighting the importance of network intrusion detection systems. This paper explores the potential and challenges of using deep reinforcement learning (DRL) in network intrusion detection. It begins by introducing key DRL concepts and frameworks, such as deep Q-networks and actor-critic algorithms, and reviews recent research utilizing DRL for intrusion detection. The study evaluates challenges related to model training efficiency, detection of minority and unknown class attacks, feature selection, and handling unbalanced datasets. The performance of DRL models is comprehensively analyzed, showing that while DRL holds promise, many recent technologies remain underexplored. Some DRL models achieve state-of-the-art results on public datasets, occasionally outperforming traditional deep learning methods. The paper concludes with recommendations for enhancing DRL deployment and testing in real-world network scenarios, with a focus on Internet of Things intrusion detection. It discusses recent DRL architectures and suggests future policy functions for DRL-based intrusion detection. Finally, the paper proposes integrating DRL with generative methods to further improve performance, addressing current gaps and supporting more robust and adaptive network intrusion detection systems.

Title: Explainability of Deep Neural Networks for Brain Tumor Detection

• Authors: S.Park, J.Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07613
• Pdf URL: https://arxiv.org/pdf/2410.07613
• Copy Paste: [[2410.07613]] Explainability of Deep Neural Networks for Brain Tumor Detection(https://arxiv.org/abs/2410.07613)
• Keywords: explainability, transformer
• Abstract: Medical image classification is crucial for supporting healthcare professionals in decision-making and training. While Convolutional Neural Networks (CNNs) have traditionally dominated this field, Transformer-based models are gaining attention. In this study, we apply explainable AI (XAI) techniques to assess the performance of various models on real-world medical data and identify areas for improvement. We compare CNN models such as VGG-16, ResNet-50, and EfficientNetV2L with a Transformer model: ViT-Base-16. Our results show that data augmentation has little impact, but hyperparameter tuning and advanced modeling improve performance. CNNs, particularly VGG-16 and ResNet-50, outperform ViT-Base-16 and EfficientNetV2L, likely due to underfitting from limited data. XAI methods like LIME and SHAP further reveal that better-performing models visualize tumors more effectively. These findings suggest that CNNs with shallower architectures are more effective for small datasets and can support medical decision-making.

Title: The Plug-in Approach for Average-Reward and Discounted MDPs: Optimal Sample Complexity Analysis

• Authors: Matthew Zurek, Yudong Chen
• Subjects: cs.LG, cs.IT, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07616
• Pdf URL: https://arxiv.org/pdf/2410.07616
• Copy Paste: [[2410.07616]] The Plug-in Approach for Average-Reward and Discounted MDPs: Optimal Sample Complexity Analysis(https://arxiv.org/abs/2410.07616)
• Keywords: generative
• Abstract: We study the sample complexity of the plug-in approach for learning $\varepsilon$-optimal policies in average-reward Markov decision processes (MDPs) with a generative model. The plug-in approach constructs a model estimate then computes an average-reward optimal policy in the estimated model. Despite representing arguably the simplest algorithm for this problem, the plug-in approach has never been theoretically analyzed. Unlike the more well-studied discounted MDP reduction method, the plug-in approach requires no prior problem information or parameter tuning. Our results fill this gap and address the limitations of prior approaches, as we show that the plug-in approach is optimal in several well-studied settings without using prior knowledge. Specifically it achieves the optimal diameter- and mixing-based sample complexities of $\widetilde{O}\left(SA \frac{D}{\varepsilon^2}\right)$ and $\widetilde{O}\left(SA \frac{\tau_{\mathrm{unif}}}{\varepsilon^2}\right)$, respectively, without knowledge of the diameter $D$ or uniform mixing time $\tau_{\mathrm{unif}}$. We also obtain span-based bounds for the plug-in approach, and complement them with algorithm-specific lower bounds suggesting that they are unimprovable. Our results require novel techniques for analyzing long-horizon problems which may be broadly useful and which also improve results for the discounted plug-in approach, removing effective-horizon-related sample size restrictions and obtaining the first optimal complexity bounds for the full range of sample sizes without reward perturbation.

Title: Moyun: A Diffusion-Based Model for Style-Specific Chinese Calligraphy Generation

• Authors: Kaiyuan Liu, Jiahao Mei, Hengyu Zhang, Yihuai Zhang, Xingjiao Wu, Daoguo Dong, Liang He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07618
• Pdf URL: https://arxiv.org/pdf/2410.07618
• Copy Paste: [[2410.07618]] Moyun: A Diffusion-Based Model for Style-Specific Chinese Calligraphy Generation(https://arxiv.org/abs/2410.07618)
• Keywords: diffusion
• Abstract: Although Chinese calligraphy generation has achieved style transfer, generating calligraphy by specifying the calligrapher, font, and character style remains challenging. To address this, we propose a new Chinese calligraphy generation model 'Moyun' , which replaces the Unet in the Diffusion model with Vision Mamba and introduces the TripleLabel control mechanism to achieve controllable calligraphy generation. The model was tested on our large-scale dataset 'Mobao' of over 1.9 million images, and the results demonstrate that 'Moyun' can effectively control the generation process and produce calligraphy in the specified style. Even for calligraphy the calligrapher has not written, 'Moyun' can generate calligraphy that matches the style of the calligrapher.

Title: MorCode: Face Morphing Attack Generation using Generative Codebooks

• Authors: Aravinda Reddy PN, Raghavendra Ramachandra, Sushma Venkatesh, Krothapalli Sreenivasa Rao, Pabitra Mitra, Rakesh Krishna
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07625
• Pdf URL: https://arxiv.org/pdf/2410.07625
• Copy Paste: [[2410.07625]] MorCode: Face Morphing Attack Generation using Generative Codebooks(https://arxiv.org/abs/2410.07625)
• Keywords: attack, diffusion, generative
• Abstract: Face recognition systems (FRS) can be compromised by face morphing attacks, which blend textural and geometric information from multiple facial images. The rapid evolution of generative AI, especially Generative Adversarial Networks (GAN) or Diffusion models, where encoded images are interpolated to generate high-quality face morphing images. In this work, we present a novel method for the automatic face morphing generation method \textit{MorCode}, which leverages a contemporary encoder-decoder architecture conditioned on codebook learning to generate high-quality morphing images. Extensive experiments were performed on the newly constructed morphing dataset using five state-of-the-art morphing generation techniques using both digital and print-scan data. The attack potential of the proposed morphing generation technique, \textit{MorCode}, was benchmarked using three different face recognition systems. The obtained results indicate the highest attack potential of the proposed \textit{MorCode} when compared with five state-of-the-art morphing generation methods on both digital and print scan data.

Title: Automatic Curriculum Expert Iteration for Reliable LLM Reasoning

• Authors: Zirui Zhao, Hanze Dong, Amrita Saha, Caiming Xiong, Doyen Sahoo
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07627
• Pdf URL: https://arxiv.org/pdf/2410.07627
• Copy Paste: [[2410.07627]] Automatic Curriculum Expert Iteration for Reliable LLM Reasoning(https://arxiv.org/abs/2410.07627)
• Keywords: robust
• Abstract: Hallucinations (i.e., generating plausible but inaccurate content) and laziness (i.e. excessive refusals or defaulting to "I don't know") persist as major challenges in LLM reasoning. Current efforts to reduce hallucinations primarily focus on factual errors in knowledge-grounded tasks, often neglecting hallucinations related to faulty reasoning. Meanwhile, some approaches render LLMs overly conservative, limiting their problem-solving capabilities. To mitigate hallucination and laziness in reasoning tasks, we propose Automatic Curriculum Expert Iteration (Auto-CEI) to enhance LLM reasoning and align responses to the model's capabilities--assertively answering within its limits and declining when tasks exceed them. In our method, Expert Iteration explores the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness; it also promotes appropriate "I don't know" responses after sufficient reasoning attempts. The curriculum automatically adjusts rewards, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning its behaviour with these limits. We compare Auto-CEI with various SOTA baselines across logical reasoning, mathematics, and planning tasks, where Auto-CEI achieves superior alignment by effectively balancing assertiveness and conservativeness.

Title: Secure Wearable Apps for Remote Healthcare Through Modern Cryptography

• Authors: Andric Li, Grace Luo, Christopher Tao, Diego Zuluaga
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.07629
• Pdf URL: https://arxiv.org/pdf/2410.07629
• Copy Paste: [[2410.07629]] Secure Wearable Apps for Remote Healthcare Through Modern Cryptography(https://arxiv.org/abs/2410.07629)
• Keywords: secure, privacy, protect
• Abstract: Wearable devices like smartwatches, wristbands, and fitness trackers are designed to be lightweight devices to be worn on the human body. With the increased connectivity of wearable devices, they will become integral to remote healthcare solutions. For example, a smartwatch can measure and upload a patient's vital signs to the cloud through a network which is monitored by software backed with Artificial Intelligence. When an anomaly of a patient is detected, it will be alerted to healthcare professionals for proper intervention. Remote healthcare offers substantial benefits for both patients and healthcare providers as patients may avoid expensive in-patient care by choosing the comfort of staying at home while being monitored after a surgery and healthcare providers can resolve challenges between limited resources and a growing population. While remote healthcare through wearable devices is ubiquitous and affordable, it raises concerns about patient privacy. Patients may wonder: Is my data stored in the cloud safe? Can anyone access and manipulate my data for blackmailing? Hence, securing patient private information end-to-end becomes crucial. This paper explores solutions for applying modern cryptography to secure wearable apps and ensure patient data is protected with confidentiality, integrity, and authenticity from wearable edge to cloud.

Title: Provable Privacy Attacks on Trained Shallow Neural Networks

• Authors: Guy Smorodinsky, Gal Vardi, Itay Safran
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.07632
• Pdf URL: https://arxiv.org/pdf/2410.07632
• Copy Paste: [[2410.07632]] Provable Privacy Attacks on Trained Shallow Neural Networks(https://arxiv.org/abs/2410.07632)
• Keywords: privacy, attack, membership infer
• Abstract: We study what provable privacy attacks can be shown on trained, 2-layer ReLU neural networks. We explore two types of attacks; data reconstruction attacks, and membership inference attacks. We prove that theoretical results on the implicit bias of 2-layer neural networks can be used to provably reconstruct a set of which at least a constant fraction are training points in a univariate setting, and can also be used to identify with high probability whether a given point was used in the training set in a high dimensional setting. To the best of our knowledge, our work is the first to show provable vulnerabilities in this setting.

Title: Shift and matching queries for video semantic segmentation

• Authors: Tsubasa Mizuno, Toru Tamaki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07635
• Pdf URL: https://arxiv.org/pdf/2410.07635
• Copy Paste: [[2410.07635]] Shift and matching queries for video semantic segmentation(https://arxiv.org/abs/2410.07635)
• Keywords: segmentation
• Abstract: Video segmentation is a popular task, but applying image segmentation models frame-by-frame to videos does not preserve temporal consistency. In this paper, we propose a method to extend a query-based image segmentation model to video using feature shift and query matching. The method uses a query-based architecture, where decoded queries represent segmentation masks. These queries should be matched before performing the feature shift to ensure that the shifted queries represent the same mask across different frames. Experimental results on CityScapes-VPS and VSPW show significant improvements from the baselines, highlighting the method's effectiveness in enhancing segmentation quality while efficiently reusing pre-trained weights.

Title: FLIER: Few-shot Language Image Models Embedded with Latent Representations

• Authors: Zhinuo Zhou, Peng Zhou, Xiaoyong Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07648
• Pdf URL: https://arxiv.org/pdf/2410.07648
• Copy Paste: [[2410.07648]] FLIER: Few-shot Language Image Models Embedded with Latent Representations(https://arxiv.org/abs/2410.07648)
• Keywords: diffusion
• Abstract: As the boosting development of large vision-language models like Contrastive Language-Image Pre-training (CLIP), many CLIP-like methods have shown impressive abilities on visual recognition, especially in low-data regimes scenes. However, we have noticed that most of these methods are limited to introducing new modifications on text and image encoder. Recently, latent diffusion models (LDMs) have shown good ability on image generation. The potent capabilities of LDMs direct our focus towards the latent representations sampled by UNet. Inspired by the conjecture in CoOp that learned prompts encode meanings beyond the existing vocabulary, we assume that, for deep models, the latent representations are concise and accurate understanding of images, in which high-frequency, imperceptible details are abstracted away. In this paper, we propose a Few-shot Language Image model Embedded with latent Representations (FLIER) for image recognition by introducing a latent encoder jointly trained with CLIP's image encoder, it incorporates pre-trained vision-language knowledge of CLIP and the latent representations from Stable Diffusion. We first generate images and corresponding latent representations via Stable Diffusion with the textual inputs from GPT-3. With latent representations as "models-understandable pixels", we introduce a flexible convolutional neural network with two convolutional layers to be the latent encoder, which is simpler than most encoders in vision-language models. The latent encoder is jointly trained with CLIP's image encoder, transferring pre-trained knowledge to downstream tasks better. Experiments and extensive ablation studies on various visual classification tasks demonstrate that FLIER performs state-of-the-art on 11 datasets for most few-shot classification.

Title: StablePrompt: Automatic Prompt Tuning using Reinforcement Learning for Large Language Models

• Authors: Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, Junmo Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07652
• Pdf URL: https://arxiv.org/pdf/2410.07652
• Copy Paste: [[2410.07652]] StablePrompt: Automatic Prompt Tuning using Reinforcement Learning for Large Language Models(https://arxiv.org/abs/2410.07652)
• Keywords: large language model
• Abstract: Finding appropriate prompts for the specific task has become an important issue as the usage of Large Language Models (LLM) has expanded. Reinforcement Learning (RL) is widely used for prompt tuning, but its inherent instability and environmental dependency make it difficult to use in practice. In this paper, we propose StablePrompt, which strikes a balance between training stability and search space, mitigating the instability of RL and producing high-performance prompts. We formulate prompt tuning as an online RL problem between the agent and target LLM and introduce Adaptive Proximal Policy Optimization (APPO). APPO introduces an LLM anchor model to adaptively adjust the rate of policy updates. This allows for flexible prompt search while preserving the linguistic ability of the pre-trained LLM. StablePrompt outperforms previous methods on various tasks including text classification, question answering, and text generation. Our code can be found in github.

Title: Mechanistic Permutability: Match Features Across Layers

• Authors: Nikita Balagansky, Ian Maksimov, Daniil Gavrilov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07656
• Pdf URL: https://arxiv.org/pdf/2410.07656
• Copy Paste: [[2410.07656]] Mechanistic Permutability: Match Features Across Layers(https://arxiv.org/abs/2410.07656)
• Keywords: interpretability, data-free
• Abstract: Understanding how features evolve across layers in deep neural networks is a fundamental challenge in mechanistic interpretability, particularly due to polysemanticity and feature superposition. While Sparse Autoencoders (SAEs) have been used to extract interpretable features from individual layers, aligning these features across layers has remained an open problem. In this paper, we introduce SAE Match, a novel, data-free method for aligning SAE features across different layers of a neural network. Our approach involves matching features by minimizing the mean squared error between the folded parameters of SAEs, a technique that incorporates activation thresholds into the encoder and decoder weights to account for differences in feature scales. Through extensive experiments on the Gemma 2 language model, we demonstrate that our method effectively captures feature evolution across layers, improving feature matching quality. We also show that features persist over several layers and that our approach can approximate hidden states across layers. Our work advances the understanding of feature dynamics in neural networks and provides a new tool for mechanistic interpretability studies.

Title: SeMv-3D: Towards Semantic and Mutil-view Consistency simultaneously for General Text-to-3D Generation with Triplane Priors

• Authors: Xiao Cai, Pengpeng Zeng, Lianli Gao, Junchen Zhu, Jiaxin Zhang, Sitong Su, Heng Tao Shen, Jingkuan Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07658
• Pdf URL: https://arxiv.org/pdf/2410.07658
• Copy Paste: [[2410.07658]] SeMv-3D: Towards Semantic and Mutil-view Consistency simultaneously for General Text-to-3D Generation with Triplane Priors(https://arxiv.org/abs/2410.07658)
• Keywords: diffusion
• Abstract: Recent advancements in generic 3D content generation from text prompts have been remarkable by fine-tuning text-to-image diffusion (T2I) models or employing these T2I models as priors to learn a general text-to-3D model. While fine-tuning-based methods ensure great alignment between text and generated views, i.e., semantic consistency, their ability to achieve multi-view consistency is hampered by the absence of 3D constraints, even in limited view. In contrast, prior-based methods focus on regressing 3D shapes with any view that maintains uniformity and coherence across views, i.e., multi-view consistency, but such approaches inevitably compromise visual-textual alignment, leading to a loss of semantic details in the generated objects. To achieve semantic and multi-view consistency simultaneously, we propose SeMv-3D, a novel framework for general text-to-3d generation. Specifically, we propose a Triplane Prior Learner (TPL) that learns triplane priors with 3D spatial features to maintain consistency among different views at the 3D level, e.g., geometry and texture. Moreover, we design a Semantic-aligned View Synthesizer (SVS) that preserves the alignment between 3D spatial features and textual semantics in latent space. In SVS, we devise a simple yet effective batch sampling and rendering strategy that can generate arbitrary views in a single feed-forward inference. Extensive experiments present our SeMv-3D's superiority over state-of-the-art performances with semantic and multi-view consistency in any view. Our code and more visual results are available at this https URL.

Title: MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion

• Authors: Onkar Susladkar, Jishu Sen Gupta, Chirag Sehgal, Sparsh Mittal, Rekha Singhal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07659
• Pdf URL: https://arxiv.org/pdf/2410.07659
• Copy Paste: [[2410.07659]] MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion(https://arxiv.org/abs/2410.07659)
• Keywords: robust, diffusion, transformer
• Abstract: The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key contributions to address the challenges of spatiotemporal video processing. First, we introduce the 3D Mobile Inverted Vector-Quantization Variational Autoencoder (3D-MBQ-VAE), which combines Variational Autoencoders (VAEs) with masked token modeling to enhance spatiotemporal video compression. The model achieves superior temporal consistency and state-of-the-art (SOTA) reconstruction quality by employing a novel training strategy with full frame masking. Second, we present MotionAura, a text-to-video generation framework that utilizes vector-quantized diffusion models to discretize the latent space and capture complex motion dynamics, producing temporally coherent videos aligned with text prompts. Third, we propose a spectral transformer-based denoising network that processes video data in the frequency domain using the Fourier Transform. This method effectively captures global context and long-range dependencies for high-quality video generation and denoising. Lastly, we introduce a downstream task of Sketch Guided Video Inpainting. This task leverages Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. Our models achieve SOTA performance on a range of benchmarks. Our work offers robust frameworks for spatiotemporal modeling and user-driven video content manipulation. We will release the code, datasets, and models in open-source.

Title: Scalable and Resource-Efficient Second-Order Federated Learning via Over-the-Air Aggregation

• Authors: Abdulmomen Ghalkha, Chaouki Ben Issaid, Mehdi Bennis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07662
• Pdf URL: https://arxiv.org/pdf/2410.07662
• Copy Paste: [[2410.07662]] Scalable and Resource-Efficient Second-Order Federated Learning via Over-the-Air Aggregation(https://arxiv.org/abs/2410.07662)
• Keywords: federate
• Abstract: Second-order federated learning (FL) algorithms offer faster convergence than their first-order counterparts by leveraging curvature information. However, they are hindered by high computational and storage costs, particularly for large-scale models. Furthermore, the communication overhead associated with large models and digital transmission exacerbates these challenges, causing communication bottlenecks. In this work, we propose a scalable second-order FL algorithm using a sparse Hessian estimate and leveraging over-the-air aggregation, making it feasible for larger models. Our simulation results demonstrate more than $67\%$ of communication resources and energy savings compared to other first and second-order baselines.

Title: Invisibility Cloak: Disappearance under Human Pose Estimation via Backdoor Attacks

• Authors: Minxing Zhang, Michael Backes, Xiao Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.07670
• Pdf URL: https://arxiv.org/pdf/2410.07670
• Copy Paste: [[2410.07670]] Invisibility Cloak: Disappearance under Human Pose Estimation via Backdoor Attacks(https://arxiv.org/abs/2410.07670)
• Keywords: security, attack, robust, steal, segmentation
• Abstract: Human Pose Estimation (HPE) has been widely applied in autonomous systems such as self-driving cars. However, the potential risks of HPE to adversarial attacks have not received comparable attention with image classification or segmentation tasks. Existing works on HPE robustness focus on misleading an HPE system to provide wrong predictions that still indicate some human poses. In this paper, we study the vulnerability of HPE systems to disappearance attacks, where the attacker aims to subtly alter the HPE training process via backdoor techniques so that any input image with some specific trigger will not be recognized as involving any human pose. As humans are typically at the center of HPE systems, such attacks can induce severe security hazards, e.g., pedestrians' lives will be threatened if a self-driving car incorrectly understands the front scene due to disappearance attacks. To achieve the adversarial goal of disappearance, we propose IntC, a general framework to craft Invisibility Cloak in the HPE domain. The core of our work lies in the design of target HPE labels that do not represent any human pose. In particular, we propose three specific backdoor attacks based on our IntC framework with different label designs. IntC-S and IntC-E, respectively designed for regression- and heatmap-based HPE techniques, concentrate the keypoints of triggered images in a tiny, imperceptible region. Further, to improve the attack's stealthiness, IntC-L designs the target poisons to capture the label outputs of typical landscape images without a human involved, achieving disappearance and reducing detectability simultaneously. Extensive experiments demonstrate the effectiveness and generalizability of our IntC methods in achieving the disappearance goal. By revealing the vulnerability of HPE to disappearance and backdoor attacks, we hope our work can raise awareness of the potential risks ...

Title: MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

• Authors: Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, Zhaochun Ren
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07672
• Pdf URL: https://arxiv.org/pdf/2410.07672
• Copy Paste: [[2410.07672]] MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization(https://arxiv.org/abs/2410.07672)
• Keywords: large language model
• Abstract: As large language models (LLMs) are rapidly advancing and achieving near-human capabilities, aligning them with human values is becoming more urgent. In scenarios where LLMs outperform humans, we face a weak-to-strong alignment problem where we need to effectively align strong student LLMs through weak supervision generated by weak teachers. Existing alignment methods mainly focus on strong-to-weak alignment and self-alignment settings, and it is impractical to adapt them to the much harder weak-to-strong alignment setting. To fill this gap, we propose a multi-agent contrastive preference optimization (MACPO) framework. MACPO facilitates weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behaviors while penalizing familiar negative ones. To get this, we devise a mutual positive behavior augmentation strategy to encourage weak teachers and strong students to learn from each other's positive behavior and further provide higher quality positive behavior for the next iteration. Additionally, we propose a hard negative behavior construction strategy to induce weak teachers and strong students to generate familiar negative behavior by fine-tuning on negative behavioral data. Experimental results on the HH-RLHF and PKU-SafeRLHF datasets, evaluated using both automatic metrics and human judgments, demonstrate that MACPO simultaneously improves the alignment performance of strong students and weak teachers. Moreover, as the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance through more iteration optimization rounds.

Title: Multimodal Clickbait Detection by De-confounding Biases Using Causal Representation Inference

• Authors: Jianxing Yu, Shiqi Wang, Han Yin, Zhenlong Sun, Ruobing Xie, Bo Zhang, Yanghui Rao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07673
• Pdf URL: https://arxiv.org/pdf/2410.07673
• Copy Paste: [[2410.07673]] Multimodal Clickbait Detection by De-confounding Biases Using Causal Representation Inference(https://arxiv.org/abs/2410.07673)
• Keywords: robust
• Abstract: This paper focuses on detecting clickbait posts on the Web. These posts often use eye-catching disinformation in mixed modalities to mislead users to click for profit. That affects the user experience and thus would be blocked by content provider. To escape detection, malicious creators use tricks to add some irrelevant non-bait content into bait posts, dressing them up as legal to fool the detector. This content often has biased relations with non-bait labels, yet traditional detectors tend to make predictions based on simple co-occurrence rather than grasping inherent factors that lead to malicious behavior. This spurious bias would easily cause misjudgments. To address this problem, we propose a new debiased method based on causal inference. We first employ a set of features in multiple modalities to characterize the posts. Considering these features are often mixed up with unknown biases, we then disentangle three kinds of latent factors from them, including the invariant factor that indicates intrinsic bait intention; the causal factor which reflects deceptive patterns in a certain scenario, and non-causal noise. By eliminating the noise that causes bias, we can use invariant and causal factors to build a robust model with good generalization ability. Experiments on three popular datasets show the effectiveness of our approach.

Title: Adversarial Robustness Overestimation and Instability in TRADES

• Authors: Jonathan Weiping Li, Ren-Wei Liang, Cheng-Han Yeh, Cheng-Chang Tsai, Kuanchun Yu, Chun-Shien Lu, Shang-Tse Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07675
• Pdf URL: https://arxiv.org/pdf/2410.07675
• Copy Paste: [[2410.07675]] Adversarial Robustness Overestimation and Instability in TRADES(https://arxiv.org/abs/2410.07675)
• Keywords: attack, robust
• Abstract: This paper examines the phenomenon of probabilistic robustness overestimation in TRADES, a prominent adversarial training method. Our study reveals that TRADES sometimes yields disproportionately high PGD validation accuracy compared to the AutoAttack testing accuracy in the multiclass classification task. This discrepancy highlights a significant overestimation of robustness for these instances, potentially linked to gradient masking. We further analyze the parameters contributing to unstable models that lead to overestimation. Our findings indicate that smaller batch sizes, lower beta values (which control the weight of the robust loss term in TRADES), larger learning rates, and higher class complexity (e.g., CIFAR-100 versus CIFAR-10) are associated with an increased likelihood of robustness overestimation. By examining metrics such as the First-Order Stationary Condition (FOSC), inner-maximization, and gradient information, we identify the underlying cause of this phenomenon as gradient masking and provide insights into it. Furthermore, our experiments show that certain unstable training instances may return to a state without robust overestimation, inspiring our attempts at a solution. In addition to adjusting parameter settings to reduce instability or retraining when overestimation occurs, we recommend incorporating Gaussian noise in inputs when the FOSC score exceed the threshold. This method aims to mitigate robustness overestimation of TRADES and other similar methods at its source, ensuring more reliable representation of adversarial robustness during evaluation.

Title: Smart Audit System Empowered by LLM

• Authors: Xu Yao, Xiaoxu Wu, Xi Li, Huan Xu, Chenlei Li, Ping Huang, Si Li, Xiaoning Ma, Jiulong Shan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07677
• Pdf URL: https://arxiv.org/pdf/2410.07677
• Copy Paste: [[2410.07677]] Smart Audit System Empowered by LLM(https://arxiv.org/abs/2410.07677)
• Keywords: large language model
• Abstract: Manufacturing quality audits are pivotal for ensuring high product standards in mass production environments. Traditional auditing processes, however, are labor-intensive and reliant on human expertise, posing challenges in maintaining transparency, accountability, and continuous improvement across complex global supply chains. To address these challenges, we propose a smart audit system empowered by large language models (LLMs). Our approach introduces three innovations: a dynamic risk assessment model that streamlines audit procedures and optimizes resource allocation; a manufacturing compliance copilot that enhances data processing, retrieval, and evaluation for a self-evolving manufacturing knowledge base; and a Re-act framework commonality analysis agent that provides real-time, customized analysis to empower engineers with insights for supplier improvement. These enhancements elevate audit efficiency and effectiveness, with testing scenarios demonstrating an improvement of over 24%.

Title: FedEP: Tailoring Attention to Heterogeneous Data Distribution with Entropy Pooling for Decentralized Federated Learning

• Authors: Chao Feng, Hongjie Guan, Alberto Huertas Celdrán, Jan von der Assen, Gérôme Bovet, Burkhard Stiller
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07678
• Pdf URL: https://arxiv.org/pdf/2410.07678
• Copy Paste: [[2410.07678]] FedEP: Tailoring Attention to Heterogeneous Data Distribution with Entropy Pooling for Decentralized Federated Learning(https://arxiv.org/abs/2410.07678)
• Keywords: federate
• Abstract: Federated Learning (FL) performance is highly influenced by data distribution across clients, and non-Independent and Identically Distributed (non-IID) leads to a slower convergence of the global model and a decrease in model effectiveness. The existing algorithms for solving the non-IID problem are focused on the traditional centralized FL (CFL), where a central server is used for model aggregation. However, in decentralized FL (DFL), nodes lack the overall vision of the federation. To address the non-IID problem in DFL, this paper proposes a novel DFL aggregation algorithm, Federated Entropy Pooling (FedEP). FedEP mitigates the client drift problem by incorporating the statistical characteristics of local distributions instead of any actual data. Prior to training, each client conducts a local distribution fitting using a Gaussian Mixture Model (GMM) and shares the resulting statistical characteristics with its neighbors. After receiving the statistical characteristics shared by its neighbors, each node tries to fit the global data distribution. In the aggregation phase, each node calculates the Kullback-Leibler (KL) divergences of the local data distribution over the fitted global data distribution, giving the weights to generate the aggregated model. Extensive experiments have demonstrated that FedEP can achieve faster convergence and show higher test performance than state-of-the-art approaches.

Title: Relational Diffusion Distillation for Efficient Image Generation

• Authors: Weilun Feng, Chuanguang Yang, Zhulin An, Libo Huang, Boyu Diao, Fei Wang, Yongjun Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07679
• Pdf URL: https://arxiv.org/pdf/2410.07679
• Copy Paste: [[2410.07679]] Relational Diffusion Distillation for Efficient Image Generation(https://arxiv.org/abs/2410.07679)
• Keywords: diffusion
• Abstract: Although the diffusion model has achieved remarkable performance in the field of image generation, its high inference delay hinders its wide application in edge devices with scarce computing resources. Therefore, many training-free sampling methods have been proposed to reduce the number of sampling steps required for diffusion models. However, they perform poorly under a very small number of sampling steps. Thanks to the emergence of knowledge distillation technology, the existing training scheme methods have achieved excellent results at very low step numbers. However, the current methods mainly focus on designing novel diffusion model sampling methods with knowledge distillation. How to transfer better diffusion knowledge from teacher models is a more valuable problem but rarely studied. Therefore, we propose Relational Diffusion Distillation (RDD), a novel distillation method tailored specifically for distilling diffusion models. Unlike existing methods that simply align teacher and student models at pixel level or feature distributions, our method introduces cross-sample relationship interaction during the distillation process and alleviates the memory constraints induced by multiple sample interactions. Our RDD significantly enhances the effectiveness of the progressive distillation framework within the diffusion model. Extensive experiments on several datasets (e.g., CIFAR-10 and ImageNet) demonstrate that our proposed RDD leads to 1.47 FID decrease under 1 sampling step compared to state-of-the-art diffusion distillation methods and achieving 256x speed-up compared to DDIM strategy. Code is available at this https URL.

Title: When the Small-Loss Trick is Not Enough: Multi-Label Image Classification with Noisy Labels Applied to CCTV Sewer Inspections

• Authors: Keryan Chelouche, Marie Lachaize (VERI), Marine Bernard (VERI), Louise Olgiati, Remi Cuingnet
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07689
• Pdf URL: https://arxiv.org/pdf/2410.07689
• Copy Paste: [[2410.07689]] When the Small-Loss Trick is Not Enough: Multi-Label Image Classification with Noisy Labels Applied to CCTV Sewer Inspections(https://arxiv.org/abs/2410.07689)
• Keywords: robust
• Abstract: The maintenance of sewerage networks, with their millions of kilometers of pipe, heavily relies on efficient Closed-Circuit Television (CCTV) inspections. Many promising approaches based on multi-label image classification have leveraged databases of historical inspection reports to automate these inspections. However, the significant presence of label noise in these databases, although known, has not been addressed. While extensive research has explored the issue of label noise in singlelabel classification (SLC), little attention has been paid to label noise in multi-label classification (MLC). To address this, we first adapted three sample selection SLC methods (Co-teaching, CoSELFIE, and DISC) that have proven robust to label noise. Our findings revealed that sample selection based solely on the small-loss trick can handle complex label noise, but it is sub-optimal. Adapting hybrid sample selection methods to noisy MLC appeared to be a more promising approach. In light of this, we developed a novel method named MHSS (Multi-label Hybrid Sample Selection) based on CoSELFIE. Through an in-depth comparative study, we demonstrated the superior performance of our approach in dealing with both synthetic complex noise and real noise, thus contributing to the ongoing efforts towards effective automation of CCTV sewer pipe inspections.

Title: Growing Efficient Accurate and Robust Neural Networks on the Edge

• Authors: Vignesh Sundaresha, Naresh Shanbhag
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07691
• Pdf URL: https://arxiv.org/pdf/2410.07691
• Copy Paste: [[2410.07691]] Growing Efficient Accurate and Robust Neural Networks on the Edge(https://arxiv.org/abs/2410.07691)
• Keywords: privacy, robust
• Abstract: The ubiquitous deployment of deep learning systems on resource-constrained Edge devices is hindered by their high computational complexity coupled with their fragility to out-of-distribution (OOD) data, especially to naturally occurring common corruptions. Current solutions rely on the Cloud to train and compress models before deploying to the Edge. This incurs high energy and latency costs in transmitting locally acquired field data to the Cloud while also raising privacy concerns. We propose GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ, i.e., completely on the Edge device. Starting with a low-complexity initial backbone network, GEARnn employs One-Shot Growth (OSG) to grow a network satisfying the memory constraints of the Edge device using clean data, and robustifies the network using Efficient Robust Augmentation (ERA) to obtain the final network. We demonstrate results on a NVIDIA Jetson Xavier NX, and analyze the trade-offs between accuracy, robustness, model size, energy consumption, and training time. Our results demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.

Title: Multi-Facet Counterfactual Learning for Content Quality Evaluation

• Authors: Jiasheng Zheng, Hongyu Lin, Boxi Cao, Meng Liao, Yaojie Lu, Xianpei Han, Le Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07693
• Pdf URL: https://arxiv.org/pdf/2410.07693
• Copy Paste: [[2410.07693]] Multi-Facet Counterfactual Learning for Content Quality Evaluation(https://arxiv.org/abs/2410.07693)
• Keywords: large language model
• Abstract: Evaluating the quality of documents is essential for filtering valuable content from the current massive amount of information. Conventional approaches typically rely on a single score as a supervision signal for training content quality evaluators, which is inadequate to differentiate documents with quality variations across multiple facets. In this paper, we propose Multi-facet cOunterfactual LEarning (MOLE), a framework for efficiently constructing evaluators that perceive multiple facets of content quality evaluation. Given a specific scenario, we prompt large language models to generate counterfactual content that exhibits variations in critical quality facets compared to the original document. Furthermore, we leverage a joint training strategy based on contrastive learning and supervised learning to enable the evaluator to distinguish between different quality facets, resulting in more accurate predictions of content quality scores. Experimental results on 2 datasets across different scenarios demonstrate that our proposed MOLE framework effectively improves the correlation of document content quality evaluations with human judgments, which serve as a valuable toolkit for effective information acquisition.

Title: Test-Time Intensity Consistency Adaptation for Shadow Detection

• Authors: Leyi Zhu, Weihuang Liu, Xinyi Chen, Zimeng Li, Xuhang Chen, Zhen Wang, Chi-Man Pun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07695
• Pdf URL: https://arxiv.org/pdf/2410.07695
• Copy Paste: [[2410.07695]] Test-Time Intensity Consistency Adaptation for Shadow Detection(https://arxiv.org/abs/2410.07695)
• Keywords: robust
• Abstract: Shadow detection is crucial for accurate scene understanding in computer vision, yet it is challenged by the diverse appearances of shadows caused by variations in illumination, object geometry, and scene context. Deep learning models often struggle to generalize to real-world images due to the limited size and diversity of training datasets. To address this, we introduce TICA, a novel framework that leverages light-intensity information during test-time adaptation to enhance shadow detection accuracy. TICA exploits the inherent inconsistencies in light intensity across shadow regions to guide the model toward a more consistent prediction. A basic encoder-decoder model is initially trained on a labeled dataset for shadow detection. Then, during the testing phase, the network is adjusted for each test sample by enforcing consistent intensity predictions between two augmented input image versions. This consistency training specifically targets both foreground and background intersection regions to identify shadow regions within images accurately for robust adaptation. Extensive evaluations on the ISTD and SBU shadow detection datasets reveal that TICA significantly demonstrates that TICA outperforms existing state-of-the-art methods, achieving superior results in balanced error rate (BER).

Title: Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures

• Authors: Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, Zaiwen Wen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07698
• Pdf URL: https://arxiv.org/pdf/2410.07698
• Copy Paste: [[2410.07698]] Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures(https://arxiv.org/abs/2410.07698)
• Keywords: large language model
• Abstract: Parameter-efficient fine-tuning (PEFT) significantly reduces memory costs when adapting large language models (LLMs) for downstream applications. However, traditional first-order (FO) fine-tuning algorithms incur substantial memory overhead due to the need to store activation values for back-propagation during gradient computation, particularly in long-context fine-tuning tasks. Zeroth-order (ZO) algorithms offer a promising alternative by approximating gradients using finite differences of function values, thus eliminating the need for activation storage. Nevertheless, existing ZO methods struggle to capture the low-rank gradient structure common in LLM fine-tuning, leading to suboptimal performance. This paper proposes a low-rank ZO gradient estimator and introduces a novel low-rank ZO algorithm (LOZO) that effectively captures this structure in LLMs. We provide convergence guarantees for LOZO by framing it as a subspace optimization method. Additionally, its low-rank nature enables LOZO to integrate with momentum techniques while incurring negligible extra memory costs. Extensive experiments across various model sizes and downstream tasks demonstrate that LOZO and its momentum-based variant outperform existing ZO methods and closely approach the performance of FO algorithms.

Title: AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories

• Authors: Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, Sujian Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07706
• Pdf URL: https://arxiv.org/pdf/2410.07706
• Copy Paste: [[2410.07706]] AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories(https://arxiv.org/abs/2410.07706)
• Keywords: large language model
• Abstract: Fine-tuning on agent-environment interaction trajectory data holds significant promise for surfacing generalized agent capabilities in open-source large language models (LLMs). In this work, we introduce AgentBank, by far the largest trajectory tuning data collection featuring more than 50k diverse high-quality interaction trajectories which comprises 16 tasks covering five distinct agent skill dimensions. Leveraging a novel annotation pipeline, we are able to scale the annotated trajectories and generate a trajectory dataset with minimized difficulty bias. Furthermore, we fine-tune LLMs on AgentBank to get a series of agent models, Samoyed. Our comparative experiments demonstrate the effectiveness of scaling the interaction trajectory data to acquire generalized agent capabilities. Additional studies also reveal some key observations regarding trajectory tuning and agent skill generalization.

Title: Rethinking the Principle of Gradient Smooth Methods in Model Explanation

• Authors: Linjiang Zhou, Chao Ma, Zepeng Wang, Xiaochuan Shi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07711
• Pdf URL: https://arxiv.org/pdf/2410.07711
• Copy Paste: [[2410.07711]] Rethinking the Principle of Gradient Smooth Methods in Model Explanation(https://arxiv.org/abs/2410.07711)
• Keywords: interpretability
• Abstract: Gradient Smoothing is an efficient approach to reducing noise in gradient-based model explanation method. SmoothGrad adds Gaussian noise to mitigate much of these noise. However, the crucial hyper-parameter in this method, the variance $\sigma$ of Gaussian noise, is set manually or with heuristic approach. However, it results in the smoothed gradients still containing a certain amount of noise. In this paper, we aim to interpret SmoothGrad as a corollary of convolution, thereby re-understanding the gradient noise and the role of $\sigma$ from the perspective of confidence level. Furthermore, we propose an adaptive gradient smoothing method, AdaptGrad, based on these insights. Through comprehensive experiments, both qualitative and quantitative results demonstrate that AdaptGrad could effectively reduce almost all the noise in vanilla gradients compared with baselines methods. AdaptGrad is simple and universal, making it applicable for enhancing gradient-based interpretability methods for better visualization.

Title: On the Generalization Properties of Deep Learning for Aircraft Fuel Flow Estimation Models

• Authors: Gabriel Jarry, Ramon Dalmau, Philippe Very, Junzi Sun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07717
• Pdf URL: https://arxiv.org/pdf/2410.07717
• Copy Paste: [[2410.07717]] On the Generalization Properties of Deep Learning for Aircraft Fuel Flow Estimation Models(https://arxiv.org/abs/2410.07717)
• Keywords: robust
• Abstract: Accurately estimating aircraft fuel flow is essential for evaluating new procedures, designing next-generation aircraft, and monitoring the environmental impact of current aviation practices. This paper investigates the generalization capabilities of deep learning models in predicting fuel consumption, focusing particularly on their performance for aircraft types absent from the training data. We propose a novel methodology that integrates neural network architectures with domain generalization techniques to enhance robustness and reliability across a wide range of aircraft. A comprehensive dataset containing 101 different aircraft types, separated into training and generalization sets, with each aircraft type set containing 1,000 flights. We employed the base of aircraft data (BADA) model for fuel flow estimates, introduced a pseudo-distance metric to assess aircraft type similarity, and explored various sampling strategies to optimize model performance in data-sparse regions. Our results reveal that for previously unseen aircraft types, the introduction of noise into aircraft and engine parameters improved model generalization. The model is able to generalize with acceptable mean absolute percentage error between 2\% and 10\% for aircraft close to existing aircraft, while performance is below 1\% error for known aircraft in the training set. This study highlights the potential of combining domain-specific insights with advanced machine learning techniques to develop scalable, accurate, and generalizable fuel flow estimation models.

Title: Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation

• Authors: Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, Jingdong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07718
• Pdf URL: https://arxiv.org/pdf/2410.07718
• Copy Paste: [[2410.07718]] Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation(https://arxiv.org/abs/2410.07718)
• Keywords: diffusion, generative
• Abstract: Recent advances in latent diffusion-based generative models for portrait image animation, such as Hallo, have achieved impressive results in short-duration video synthesis. In this paper, we present updates to Hallo, introducing several design enhancements to extend its capabilities. First, we extend the method to produce long-duration videos. To address substantial challenges such as appearance drift and temporal artifacts, we investigate augmentation strategies within the image space of conditional motion frames. Specifically, we introduce a patch-drop technique augmented with Gaussian noise to enhance visual consistency and temporal coherence over long duration. Second, we achieve 4K resolution portrait video generation. To accomplish this, we implement vector quantization of latent codes and apply temporal alignment techniques to maintain coherence across the temporal dimension. By integrating a high-quality decoder, we realize visual synthesis at 4K resolution. Third, we incorporate adjustable semantic textual labels for portrait expressions as conditional inputs. This extends beyond traditional audio cues to improve controllability and increase the diversity of the generated content. To the best of our knowledge, Hallo2, proposed in this paper, is the first method to achieve 4K resolution and generate hour-long, audio-driven portrait image animations enhanced with textual prompts. We have conducted extensive experiments to evaluate our method on publicly available datasets, including HDTF, CelebV, and our introduced "Wild" dataset. The experimental results demonstrate that our approach achieves state-of-the-art performance in long-duration portrait video animation, successfully generating rich and controllable content at 4K resolution for duration extending up to tens of minutes. Project page this https URL

Title: Understanding Adversarially Robust Generalization via Weight-Curvature Index

• Authors: Yuelin Xu, Xiao Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07719
• Pdf URL: https://arxiv.org/pdf/2410.07719
• Copy Paste: [[2410.07719]] Understanding Adversarially Robust Generalization via Weight-Curvature Index(https://arxiv.org/abs/2410.07719)
• Keywords: security, robust
• Abstract: Despite extensive research on adversarial examples, the underlying mechanisms of adversarially robust generalization, a critical yet challenging task for deep learning, remain largely unknown. In this work, we propose a novel perspective to decipher adversarially robust generalization through the lens of the Weight-Curvature Index (WCI). The proposed WCI quantifies the vulnerability of models to adversarial perturbations using the Frobenius norm of weight matrices and the trace of Hessian matrices. We prove generalization bounds based on PAC-Bayesian theory and second-order loss function approximations to elucidate the interplay between robust generalization gap, model parameters, and loss landscape curvature. Our theory and experiments show that WCI effectively captures the robust generalization performance of adversarially trained models. By offering a nuanced understanding of adversarial robustness based on the scale of model parameters and the curvature of the loss landscape, our work provides crucial insights for designing more resilient deep learning models, enhancing their reliability and security.

Title: Towards Trustworthy Web Attack Detection: An Uncertainty-Aware Ensemble Deep Kernel Learning Model

• Authors: Yonghang Zhou, Hongyi Zhu, Yidong Chai, Yuanchun Jiang, Yezheng Liu
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2410.07725
• Pdf URL: https://arxiv.org/pdf/2410.07725
• Copy Paste: [[2410.07725]] Towards Trustworthy Web Attack Detection: An Uncertainty-Aware Ensemble Deep Kernel Learning Model(https://arxiv.org/abs/2410.07725)
• Keywords: attack
• Abstract: Web attacks are one of the major and most persistent forms of cyber threats, which bring huge costs and losses to web application-based businesses. Various detection methods, such as signature-based, machine learning-based, and deep learning-based, have been proposed to identify web attacks. However, these methods either (1) heavily rely on accurate and complete rule design and feature engineering, which may not adapt to fast-evolving attacks, or (2) fail to estimate model uncertainty, which is essential to the trustworthiness of the prediction made by the model. In this study, we proposed an Uncertainty-aware Ensemble Deep Kernel Learning (UEDKL) model to detect web attacks from HTTP request payload data with the model uncertainty captured from the perspective of both data distribution and model parameters. The proposed UEDKL utilizes a deep kernel learning model to distinguish normal HTTP requests from different types of web attacks with model uncertainty estimated from data distribution perspective. Multiple deep kernel learning models were trained as base learners to capture the model uncertainty from model parameters perspective. An attention-based ensemble learning approach was designed to effectively integrate base learners' predictions and model uncertainty. We also proposed a new metric named High Uncertainty Ratio-F Score Curve to evaluate model uncertainty estimation. Experiments on BDCI and SRBH datasets demonstrated that the proposed UEDKL framework yields significant improvement in both web attack detection performance and uncertainty estimation quality compared to benchmark models.

Title: Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning

• Authors: Jingyuan Zhang, Yiyang Duan, Shuaicheng Niu, Yang Cao, Wei Yang Bryan Lim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07738
• Pdf URL: https://arxiv.org/pdf/2410.07738
• Copy Paste: [[2410.07738]] Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning(https://arxiv.org/abs/2410.07738)
• Keywords: privacy, protect, attack, robust, federate
• Abstract: Federated Domain Adaptation (FDA) is a Federated Learning (FL) scenario where models are trained across multiple clients with unique data domains but a shared category space, without transmitting private data. The primary challenge in FDA is data heterogeneity, which causes significant divergences in gradient updates when using conventional averaging-based aggregation methods, reducing the efficacy of the global model. This further undermines both in-domain and out-of-domain performance (within the same federated system but outside the local client). To address this, we propose a novel framework called \textbf{M}ulti-domain \textbf{P}rototype-based \textbf{F}ederated Fine-\textbf{T}uning (MPFT). MPFT fine-tunes a pre-trained model using multi-domain prototypes, i.e., pretrained representations enriched with domain-specific information from category-specific local data. This enables supervised learning on the server to derive a globally optimized adapter that is subsequently distributed to local clients, without the intrusion of data privacy. Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation in FDA. Notably, MPFT achieves convergence within a single communication round, greatly reducing computation and communication costs. To ensure privacy, MPFT applies differential privacy to protect the prototypes. Additionally, we develop a prototype-based feature space hijacking attack to evaluate robustness, confirming that raw data samples remain unrecoverable even after extensive training epochs. The complete implementation of MPFL is available at \url{this https URL}.

Title: StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs

• Authors: Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai Wang, Chuhan Wu, Zhiqiang Guo, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07745
• Pdf URL: https://arxiv.org/pdf/2410.07745
• Copy Paste: [[2410.07745]] StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs(https://arxiv.org/abs/2410.07745)
• Keywords: robust, large language model
• Abstract: Despite having powerful reasoning and inference capabilities, Large Language Models (LLMs) still need external tools to acquire real-time information retrieval or domain-specific expertise to solve complex tasks, which is referred to as tool learning. Existing tool learning methods primarily rely on tuning with expert trajectories, focusing on token-sequence learning from a linguistic perspective. However, there are several challenges: 1) imitating static trajectories limits their ability to generalize to new tasks. 2) even expert trajectories can be suboptimal, and better solution paths may exist. In this work, we introduce StepTool, a novel step-grained reinforcement learning framework to improve tool learning in LLMs. It consists of two components: Step-grained Reward Shaping, which assigns rewards at each tool interaction based on tool invocation success and its contribution to the task, and Step-grained Optimization, which uses policy gradient methods to optimize the model in a multi-step manner. Experimental results demonstrate that StepTool significantly outperforms existing methods in multi-step, tool-based tasks, providing a robust solution for complex task environments. Codes are available at this https URL.

Title: Benign Overfitting in Single-Head Attention

• Authors: Roey Magen, Shuning Shang, Zhiwei Xu, Spencer Frei, Wei Hu, Gal Vardi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07746
• Pdf URL: https://arxiv.org/pdf/2410.07746
• Copy Paste: [[2410.07746]] Benign Overfitting in Single-Head Attention(https://arxiv.org/abs/2410.07746)
• Keywords: transformer
• Abstract: The phenomenon of benign overfitting, where a trained neural network perfectly fits noisy training data but still achieves near-optimal test performance, has been extensively studied in recent years for linear models and fully-connected/convolutional networks. In this work, we study benign overfitting in a single-head softmax attention model, which is the fundamental building block of Transformers. We prove that under appropriate conditions, the model exhibits benign overfitting in a classification setting already after two steps of gradient descent. Moreover, we show conditions where a minimum-norm/maximum-margin interpolator exhibits benign overfitting. We study how the overfitting behavior depends on the signal-to-noise ratio (SNR) of the data distribution, namely, the ratio between norms of signal and noise tokens, and prove that a sufficiently large SNR is both necessary and sufficient for benign overfitting.

Title: TVBench: Redesigning Video-Language Evaluation

• Authors: Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees G. M. Snoek, Yuki M. Asano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07752
• Pdf URL: https://arxiv.org/pdf/2410.07752
• Copy Paste: [[2410.07752]] TVBench: Redesigning Video-Language Evaluation(https://arxiv.org/abs/2410.07752)
• Keywords: large language model
• Abstract: Large language models have demonstrated impressive performance when integrated with vision models even enabling video understanding. However, evaluating these video models presents its own unique challenges, for which several benchmarks have been proposed. In this paper, we show that the currently most used video-language benchmarks can be solved without requiring much temporal reasoning. We identified three main issues in existing datasets: (i) static information from single frames is often sufficient to solve the tasks (ii) the text of the questions and candidate answers is overly informative, allowing models to answer correctly without relying on any visual input (iii) world knowledge alone can answer many of the questions, making the benchmarks a test of knowledge replication rather than visual reasoning. In addition, we found that open-ended question-answering benchmarks for video understanding suffer from similar issues while the automatic evaluation process with LLMs is unreliable, making it an unsuitable alternative. As a solution, we propose TVBench, a novel open-source video multiple-choice question-answering benchmark, and demonstrate through extensive evaluations that it requires a high level of temporal understanding. Surprisingly, we find that most recent state-of-the-art video-language models perform similarly to random performance on TVBench, with only Gemini-Pro and Tarsier clearly surpassing this baseline.

Title: Synthesizing Multi-Class Surgical Datasets with Anatomy-Aware Diffusion Models

• Authors: Danush Kumar Venkatesh, Dominik Rivoir, Micha Pfeiffer, Fiona Kolbinger, Stefanie Speidel
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07753
• Pdf URL: https://arxiv.org/pdf/2410.07753
• Copy Paste: [[2410.07753]] Synthesizing Multi-Class Surgical Datasets with Anatomy-Aware Diffusion Models(https://arxiv.org/abs/2410.07753)
• Keywords: diffusion, segmentation
• Abstract: In computer-assisted surgery, automatically recognizing anatomical organs is crucial for understanding the surgical scene and providing intraoperative assistance. While machine learning models can identify such structures, their deployment is hindered by the need for labeled, diverse surgical datasets with anatomical annotations. Labeling multiple classes (i.e., organs) in a surgical scene is time-intensive, requiring medical experts. Although synthetically generated images can enhance segmentation performance, maintaining both organ structure and texture during generation is challenging. We introduce a multi-stage approach using diffusion models to generate multi-class surgical datasets with annotations. Our framework improves anatomy awareness by training organ specific models with an inpainting objective guided by binary segmentation masks. The organs are generated with an inference pipeline using pre-trained ControlNet to maintain the organ structure. The synthetic multi-class datasets are constructed through an image composition step, ensuring structural and textural consistency. This versatile approach allows the generation of multi-class datasets from real binary datasets and simulated surgical masks. We thoroughly evaluate the generated datasets on image quality and downstream segmentation, achieving a $15\%$ improvement in segmentation scores when combined with real images. Our codebase this https URL

Title: HeightFormer: A Semantic Alignment Monocular 3D Object Detection Method from Roadside Perspective

• Authors: Pei Liu (1), Zihao Zhang (2), Haipeng Liu (3), Nanfang Zheng (4), Meixin Zhu (1), Ziyuan Pu (4) ((1) Intelligent Transportation Thrust, Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), (2) School of Cyber Science and Engineering, Southeast University, (3) Li Auto Inc, (4) School of Transportation, Southeast University)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07758
• Pdf URL: https://arxiv.org/pdf/2410.07758
• Copy Paste: [[2410.07758]] HeightFormer: A Semantic Alignment Monocular 3D Object Detection Method from Roadside Perspective(https://arxiv.org/abs/2410.07758)
• Keywords: robust, extraction
• Abstract: The on-board 3D object detection technology has received extensive attention as a critical technology for autonomous driving, while few studies have focused on applying roadside sensors in 3D traffic object detection. Existing studies achieve the projection of 2D image features to 3D features through height estimation based on the frustum. However, they did not consider the height alignment and the extraction efficiency of bird's-eye-view features. We propose a novel 3D object detection framework integrating Spatial Former and Voxel Pooling Former to enhance 2D-to-3D projection based on height estimation. Extensive experiments were conducted using the Rope3D and DAIR-V2X-I dataset, and the results demonstrated the outperformance of the proposed algorithm in the detection of both vehicles and cyclists. These results indicate that the algorithm is robust and generalized under various detection scenarios. Improving the accuracy of 3D object detection on the roadside is conducive to building a safe and trustworthy intelligent transportation system of vehicle-road coordination and promoting the large-scale application of autonomous driving. The code and pre-trained models will be released on this https URL.

Title: $\textit{Jump Your Steps}$: Optimizing Sampling Schedule of Discrete Diffusion Models

• Authors: Yong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Yuki Mitsufuji
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07761
• Pdf URL: https://arxiv.org/pdf/2410.07761
• Copy Paste: [[2410.07761]] $\textit{Jump Your Steps}$: Optimizing Sampling Schedule of Discrete Diffusion Models(https://arxiv.org/abs/2410.07761)
• Keywords: diffusion
• Abstract: Diffusion models have seen notable success in continuous domains, leading to the development of discrete diffusion models (DDMs) for discrete variables. Despite recent advances, DDMs face the challenge of slow sampling speeds. While parallel sampling methods like $\tau$-leaping accelerate this process, they introduce $\textit{Compounding Decoding Error}$ (CDE), where discrepancies arise between the true distribution and the approximation from parallel token generation, leading to degraded sample quality. In this work, we present $\textit{Jump Your Steps}$ (JYS), a novel approach that optimizes the allocation of discrete sampling timesteps by minimizing CDE without extra computational cost. More precisely, we derive a practical upper bound on CDE and propose an efficient algorithm for searching for the optimal sampling schedule. Extensive experiments across image, music, and text generation show that JYS significantly improves sampling quality, establishing it as a versatile framework for enhancing DDM performance for fast sampling.

Title: HARIVO: Harnessing Text-to-Image Models for Video Generation

• Authors: Mingi Kwon, Seoung Wug Oh, Yang Zhou, Difan Liu, Joon-Young Lee, Haoran Cai, Baqiao Liu, Feng Liu, Youngjung Uh
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07763
• Pdf URL: https://arxiv.org/pdf/2410.07763
• Copy Paste: [[2410.07763]] HARIVO: Harnessing Text-to-Image Models for Video Generation(https://arxiv.org/abs/2410.07763)
• Keywords: diffusion
• Abstract: We present a method to create diffusion-based video models from pretrained Text-to-Image (T2I) models. Recently, AnimateDiff proposed freezing the T2I model while only training temporal layers. We advance this method by proposing a unique architecture, incorporating a mapping network and frame-wise tokens, tailored for video generation while maintaining the diversity and creativity of the original T2I model. Key innovations include novel loss functions for temporal smoothness and a mitigating gradient sampling technique, ensuring realistic and temporally consistent video generation despite limited public video data. We have successfully integrated video-specific inductive biases into the architecture and loss functions. Our method, built on the frozen StableDiffusion model, simplifies training processes and allows for seamless integration with off-the-shelf models like ControlNet and DreamBooth. project page: this https URL

Title: Explaining Hypergraph Neural Networks: From Local Explanations to Global Concepts

• Authors: Shiye Su, Iulia Duta, Lucie Charlotte Magister, Pietro Liò
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07764
• Pdf URL: https://arxiv.org/pdf/2410.07764
• Copy Paste: [[2410.07764]] Explaining Hypergraph Neural Networks: From Local Explanations to Global Concepts(https://arxiv.org/abs/2410.07764)
• Keywords: extraction, explainability
• Abstract: Hypergraph neural networks are a class of powerful models that leverage the message passing paradigm to learn over hypergraphs, a generalization of graphs well-suited to describing relational data with higher-order interactions. However, such models are not naturally interpretable, and their explainability has received very limited attention. We introduce SHypX, the first model-agnostic post-hoc explainer for hypergraph neural networks that provides both local and global explanations. At the instance-level, it performs input attribution by discretely sampling explanation subhypergraphs optimized to be faithful and concise. At the model-level, it produces global explanation subhypergraphs using unsupervised concept extraction. Extensive experiments across four real-world and four novel, synthetic hypergraph datasets demonstrate that our method finds high-quality explanations which can target a user-specified balance between faithfulness and concision, improving over baselines by 25 percent points in fidelity on average.

Title: GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps

• Authors: Muhammad Umair Nasir, Steven James, Julian Togelius
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07765
• Pdf URL: https://arxiv.org/pdf/2410.07765
• Copy Paste: [[2410.07765]] GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps(https://arxiv.org/abs/2410.07765)
• Keywords: large language model
• Abstract: Large language models (LLMs) have recently demonstrated great success in generating and understanding natural language. While they have also shown potential beyond the domain of natural language, it remains an open question as to what extent and in which way these LLMs can plan. We investigate their planning capabilities by proposing GameTraversalBenchmark (GTB), a benchmark consisting of diverse 2D grid-based game maps. An LLM succeeds if it can traverse through given objectives, with a minimum number of steps and a minimum number of generation errors. We evaluate a number of LLMs on GTB and found that GPT-4-Turbo achieved the highest score of 44.97% on GTB\_Score (GTBS), a composite score that combines the three above criteria. Furthermore, we preliminarily test large reasoning models, namely o1, which scores $67.84\%$ on GTBS, indicating that the benchmark remains challenging for current models. Code, data, and documentation are available at this https URL.

Title: Dialectical Behavior Therapy Approach to LLM Prompting

• Authors: Oxana Vitman, Nika Amaglobeli, Paul Plachinda
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07768
• Pdf URL: https://arxiv.org/pdf/2410.07768
• Copy Paste: [[2410.07768]] Dialectical Behavior Therapy Approach to LLM Prompting(https://arxiv.org/abs/2410.07768)
• Keywords: large language model
• Abstract: Large language models demonstrated state-of-the-art results on various reasoning tasks when applying the chain-of-thought (CoT) prompting technique. CoT prompting guides the model into breaking tasks into a few intermediate steps and provides step-by-step demonstrations. However, solving complex reasoning tasks remains a challenge. In this paper, we propose a novel prompting strategy inspired by Dialectical Behavioral Therapy (DBT). DBT, a form of cognitive-behavioral therapy, aims to help individuals cope with stress by developing a system of reasoning. We applied DBT's basic concepts of shaping dialog to construct prompts and conducted experiments on different datasets and LLMs with various numbers of parameters. Our results show that prompts crafted with DBT techniques significantly improve results on smaller models, achieving a 7% increase in accuracy on the StrategyQA, 4.8% on Aqua dataset using 8b parameters model, and a 16.2% increase on the StrategyQA, 5.3% on GSM8K dataset with 14b parameters model.

Title: Towards Quantifying The Privacy Of Redacted Text

• Authors: Vaibhav Gusain, Douglas Leith
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07772
• Pdf URL: https://arxiv.org/pdf/2410.07772
• Copy Paste: [[2410.07772]] Towards Quantifying The Privacy Of Redacted Text(https://arxiv.org/abs/2410.07772)
• Keywords: privacy, transformer
• Abstract: In this paper we propose use of a k-anonymity-like approach for evaluating the privacy of redacted text. Given a piece of redacted text we use a state of the art transformer-based deep learning network to reconstruct the original text. This generates multiple full texts that are consistent with the redacted text, i.e. which are grammatical, have the same non-redacted words etc, and represents each of these using an embedding vector that captures sentence similarity. In this way we can estimate the number, diversity and quality of full text consistent with the redacted text and so evaluate privacy.

Title: Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation

• Authors: Sweta Agrawal, José G. C. de Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno M Guerreiro, Andre Martins
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07779
• Pdf URL: https://arxiv.org/pdf/2410.07779
• Copy Paste: [[2410.07779]] Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation(https://arxiv.org/abs/2410.07779)
• Keywords: large language model
• Abstract: Alignment with human preferences is an important step in developing accurate and safe large language models. This is no exception in machine translation (MT), where better handling of language nuances and context-specific variations leads to improved quality. However, preference data based on human feedback can be very expensive to obtain and curate at a large scale. Automatic metrics, on the other hand, can induce preferences, but they might not match human expectations perfectly. In this paper, we propose an approach that leverages the best of both worlds. We first collect sentence-level quality assessments from professional linguists on translations generated by multiple high-quality MT systems and evaluate the ability of current automatic metrics to recover these preferences. We then use this analysis to curate a new dataset, MT-Pref (metric induced translation preference) dataset, which comprises 18k instances covering 18 language directions, using texts sourced from multiple domains post-2022. We show that aligning TOWER models on MT-Pref significantly improves translation quality on WMT23 and FLORES benchmarks.

Title: Rewriting Conversational Utterances with Instructed Large Language Models

• Authors: Elnara Galimzhanova, Cristina Ioana Muntean, Franco Maria Nardini, Raffaele Perego, Guido Rocchietti
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.07797
• Pdf URL: https://arxiv.org/pdf/2410.07797
• Copy Paste: [[2410.07797]] Rewriting Conversational Utterances with Instructed Large Language Models(https://arxiv.org/abs/2410.07797)
• Keywords: large language model
• Abstract: Many recent studies have shown the ability of large language models (LLMs) to achieve state-of-the-art performance on many NLP tasks, such as question answering, text summarization, coding, and translation. In some cases, the results provided by LLMs are on par with those of human experts. These models' most disruptive innovation is their ability to perform tasks via zero-shot or few-shot prompting. This capability has been successfully exploited to train instructed LLMs, where reinforcement learning with human feedback is used to guide the model to follow the user's requests directly. In this paper, we investigate the ability of instructed LLMs to improve conversational search effectiveness by rewriting user questions in a conversational setting. We study which prompts provide the most informative rewritten utterances that lead to the best retrieval performance. Reproducible experiments are conducted on publicly-available TREC CAST datasets. The results show that rewriting conversational utterances with instructed LLMs achieves significant improvements of up to 25.2% in MRR, 31.7% in Precision@1, 27% in NDCG@3, and 11.5% in Recall@500 over state-of-the-art techniques.

Title: Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Transformers

• Authors: Alireza Naderi, Thiziri Nait Saada, Jared Tanner
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.07799
• Pdf URL: https://arxiv.org/pdf/2410.07799
• Copy Paste: [[2410.07799]] Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Transformers(https://arxiv.org/abs/2410.07799)
• Keywords: transformer
• Abstract: Attention layers are the core component of transformers, the current state-of-the-art neural network architecture. However, \softmaxx-based attention puts transformers' trainability at risk. Even \textit{at initialisation}, the propagation of signals and gradients through the random network can be pathological, resulting in known issues such as (i) vanishing/exploding gradients and (ii) \textit{rank collapse}, i.e. when all tokens converge to a single representation \textit{with depth}. This paper examines signal propagation in \textit{attention-only} transformers from a random matrix perspective, illuminating the origin of such issues, as well as unveiling a new phenomenon -- (iii) rank collapse \textit{in width}. Modelling \softmaxx-based attention at initialisation with Random Markov matrices, our theoretical analysis reveals that a \textit{spectral gap} between the two largest singular values of the attention matrix causes (iii), which, in turn, exacerbates (i) and (ii). Building on this insight, we propose a novel, yet simple, practical solution to resolve rank collapse in width by removing the spectral gap. Moreover, we validate our findings and discuss the training benefits of the proposed fix through experiments that also motivate a revision of some of the default parameter scaling. Our attention model accurately describes the standard key-query attention in a single-layer transformer, making this work a significant first step towards a better understanding of the initialisation dynamics in the multi-layer case.

Title: MGMD-GAN: Generalization Improvement of Generative Adversarial Networks with Multiple Generator Multiple Discriminator Framework Against Membership Inference Attacks

• Authors: Nirob Arefin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07803
• Pdf URL: https://arxiv.org/pdf/2410.07803
• Copy Paste: [[2410.07803]] MGMD-GAN: Generalization Improvement of Generative Adversarial Networks with Multiple Generator Multiple Discriminator Framework Against Membership Inference Attacks(https://arxiv.org/abs/2410.07803)
• Keywords: attack, membership infer, generative
• Abstract: Generative Adversarial Networks (GAN) are among the widely used Generative models in various applications. However, the original GAN architecture may memorize the distribution of the training data and, therefore, poses a threat to Membership Inference Attacks. In this work, we propose a new GAN framework that consists of Multiple Generators and Multiple Discriminators (MGMD-GAN). Disjoint partitions of the training data are used to train this model and it learns the mixture distribution of all the training data partitions. In this way, our proposed model reduces the generalization gap which makes our MGMD-GAN less vulnerable to Membership Inference Attacks. We provide an experimental analysis of our model and also a comparison with other GAN frameworks.

Title: Towards Robust IoT Defense: Comparative Statistics of Attack Detection in Resource-Constrained Scenarios

• Authors: Zainab Alwaisi, Simone Soderi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.07810
• Pdf URL: https://arxiv.org/pdf/2410.07810
• Copy Paste: [[2410.07810]] Towards Robust IoT Defense: Comparative Statistics of Attack Detection in Resource-Constrained Scenarios(https://arxiv.org/abs/2410.07810)
• Keywords: security, defense, attack, robust
• Abstract: Resource constraints pose a significant cybersecurity threat to IoT smart devices, making them vulnerable to various attacks, including those targeting energy and memory. This study underscores the need for innovative security measures due to resource-related incidents in smart devices. In this paper, we conduct an extensive statistical analysis of cyberattack detection algorithms under resource constraints to identify the most efficient one. Our research involves a comparative analysis of various algorithms, including those from our previous work. We specifically compare a lightweight algorithm for detecting resource-constrained cyberattacks with another designed for the same purpose. The latter employs TinyML for detection. In addition to the comprehensive evaluation of the proposed algorithms, we introduced a novel detection method for resource-constrained attacks. This method involves analyzing protocol data and categorizing the final data packet as normal or attacked. The attacked data is further analyzed in terms of the memory and energy consumption of the devices to determine whether it is an energy or memory attack or another form of malicious activity. We compare the suggested algorithm performance using four evaluation metrics: accuracy, PoD, PoFA, and PoM. The proposed dynamic techniques dynamically select the classifier with the best results for detecting attacks, ensuring optimal performance even within resource-constrained IoT environments. The results indicate that the proposed algorithms outperform the existing works with accuracy for algorithms with TinyML and without TinyML of 99.3\%, 98.2\%, a probability of detection of 99.4\%, 97.3\%, a probability of false alarm of 1.23\%, 1.64\%, a probability of misdetection of 1.64\%, 1.46 respectively. In contrast, the accuracy of the novel detection mechanism exceeds 99.5\% for RF and 97\% for SVM.

Title: Simple ReFlow: Improved Techniques for Fast Flow Models

• Authors: Beomsu Kim, Yu-Guan Hsieh, Michal Klein, Marco Cuturi, Jong Chul Ye, Bahjat Kawar, James Thornton
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07815
• Pdf URL: https://arxiv.org/pdf/2410.07815
• Copy Paste: [[2410.07815]] Simple ReFlow: Improved Techniques for Fast Flow Models(https://arxiv.org/abs/2410.07815)
• Keywords: diffusion, generative
• Abstract: Diffusion and flow-matching models achieve remarkable generative performance but at the cost of many sampling steps, this slows inference and limits applicability to time-critical tasks. The ReFlow procedure can accelerate sampling by straightening generation trajectories. However, ReFlow is an iterative procedure, typically requiring training on simulated data, and results in reduced sample quality. To mitigate sample deterioration, we examine the design space of ReFlow and highlight potential pitfalls in prior heuristic practices. We then propose seven improvements for training dynamics, learning and inference, which are verified with thorough ablation studies on CIFAR10 $32 \times 32$, AFHQv2 $64 \times 64$, and FFHQ $64 \times 64$. Combining all our techniques, we achieve state-of-the-art FID scores (without / with guidance, resp.) for fast generation via neural ODEs: $2.23$ / $1.98$ on CIFAR10, $2.30$ / $1.91$ on AFHQv2, $2.84$ / $2.67$ on FFHQ, and $3.49$ / $1.74$ on ImageNet-64, all with merely $9$ neural function evaluations.

Title: Uncovering Overfitting in Large Language Model Editing

• Authors: Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, Zhumin Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07819
• Pdf URL: https://arxiv.org/pdf/2410.07819
• Copy Paste: [[2410.07819]] Uncovering Overfitting in Large Language Model Editing(https://arxiv.org/abs/2410.07819)
• Keywords: large language model
• Abstract: Knowledge editing has been proposed as an effective method for updating and correcting the internal knowledge of Large Language Models (LLMs). However, existing editing methods often struggle with complex tasks, such as multi-hop reasoning. In this paper, we identify and investigate the phenomenon of Editing Overfit, where edited models assign disproportionately high probabilities to the edit target, hindering the generalization of new knowledge in complex scenarios. We attribute this issue to the current editing paradigm, which places excessive emphasis on the direct correspondence between the input prompt and the edit target for each edit sample. To further explore this issue, we introduce a new benchmark, EVOKE (EValuation of Editing Overfit in Knowledge Editing), along with fine-grained evaluation metrics. Through comprehensive experiments and analysis, we demonstrate that Editing Overfit is prevalent in current editing methods and that common overfitting mitigation strategies are of limited effectiveness in knowledge editing. To overcome this, inspired by LLMs' knowledge recall mechanisms, we propose a new plug-and-play strategy called Learn to Inference (LTI), which introduce a Multi-stage Inference Constraint module to guide the edited models in recalling new knowledge similarly to how unedited LLMs leverage knowledge through in-context learning. Extensive experimental results across a wide range of tasks validate the effectiveness of LTI in mitigating Editing Overfit.

Title: Exploring Foundation Models in Remote Sensing Image Change Detection: A Comprehensive Survey

• Authors: Zihan Yu, Tianxiao Li, Yuxin Zhu, Rongze Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07824
• Pdf URL: https://arxiv.org/pdf/2410.07824
• Copy Paste: [[2410.07824]] Exploring Foundation Models in Remote Sensing Image Change Detection: A Comprehensive Survey(https://arxiv.org/abs/2410.07824)
• Keywords: extraction
• Abstract: Change detection, as an important and widely applied technique in the field of remote sensing, aims to analyze changes in surface areas over time and has broad applications in areas such as environmental monitoring, urban development, and land use this http URL recent years, deep learning, especially the development of foundation models, has provided more powerful solutions for feature extraction and data fusion, effectively addressing these complexities. This paper systematically reviews the latest advancements in the field of change detection, with a focus on the application of foundation models in remote sensing tasks.

Title: Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models

• Authors: Zhipeng Chen, Liang Song, Kun Zhou, Wayne Xin Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07825
• Pdf URL: https://arxiv.org/pdf/2410.07825
• Copy Paste: [[2410.07825]] Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models(https://arxiv.org/abs/2410.07825)
• Keywords: extraction, large language model
• Abstract: Multi-lingual ability transfer has become increasingly important for the broad application of large language models (LLMs). Existing work highly relies on training with the multi-lingual ability-related data, which may be not available for low-resource languages. To solve it, we propose a Multi-lingual Ability Extraction and Transfer approach, named as MAET. Our key idea is to decompose and extract language-agnostic ability-related weights from LLMs, and transfer them across different languages by simple addition and subtraction operations without training. Specially, our MAET consists of the extraction and transfer stages. In the extraction stage, we firstly locate key neurons that are highly related to specific abilities, and then employ them to extract the transferable ability-specific weights. In the transfer stage, we further select the ability-related parameter tensors, and design the merging strategy based on the linguistic and ability specific weights, to build the multi-lingual ability-enhanced LLM. To demonstrate the effectiveness of our proposed approach, we conduct extensive experiments on mathematical and scientific tasks in both high-resource lingual and low-resource lingual scenarios. Experiment results have shown that MAET can effectively and efficiently extract and transfer the advanced abilities, and outperform training-based baseline methods. Our code and data are available at \url{this https URL}.

Title: NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Models

• Authors: William Tan, Kevin Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07830
• Pdf URL: https://arxiv.org/pdf/2410.07830
• Copy Paste: [[2410.07830]] NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Models(https://arxiv.org/abs/2410.07830)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional promise in translation tasks for high-resource languages. However, their performance in low-resource languages is limited by the scarcity of both parallel and monolingual corpora, as well as the presence of noise. Consequently, such LLMs suffer with alignment and have lagged behind State-of-The-Art (SoTA) neural machine translation (NMT) models in these settings. This paper introduces NusaMT-7B, an LLM-based machine translation model for low-resource Indonesian languages, starting with Balinese and Minangkabau. Leveraging the pretrained LLaMA2-7B, our approach integrates continued pre-training on monolingual data, Supervised Fine-Tuning (SFT), self-learning, and an LLM-based data cleaner to reduce noise in parallel sentences. In the FLORES-200 multilingual translation benchmark, NusaMT-7B outperforms SoTA models in the spBLEU metric by up to +6.69 spBLEU in translations into Balinese and Minangkabau, but underperforms by up to -3.38 spBLEU in translations into higher-resource languages. Our results show that fine-tuned LLMs can enhance translation quality for low-resource languages, aiding in linguistic preservation and cross-cultural communication.

Title: Multi-Scale Deformable Transformers for Student Learning Behavior Detection in Smart Classroom

• Authors: Zhifeng Wang, Minghui Wang, Chunyan Zeng, Longlong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07834
• Pdf URL: https://arxiv.org/pdf/2410.07834
• Copy Paste: [[2410.07834]] Multi-Scale Deformable Transformers for Student Learning Behavior Detection in Smart Classroom(https://arxiv.org/abs/2410.07834)
• Keywords: robust, extraction, transformer
• Abstract: The integration of Artificial Intelligence into the modern educational system is rapidly evolving, particularly in monitoring student behavior in classrooms, a task traditionally dependent on manual observation. This conventional method is notably inefficient, prompting a shift toward more advanced solutions like computer vision. However, existing target detection models face significant challenges such as occlusion, blurring, and scale disparity, which are exacerbated by the dynamic and complex nature of classroom settings. Furthermore, these models must adeptly handle multiple target detection. To overcome these obstacles, we introduce the Student Learning Behavior Detection with Multi-Scale Deformable Transformers (SCB-DETR), an innovative approach that utilizes large convolutional kernels for upstream feature extraction, and multi-scale feature fusion. This technique significantly improves the detection capabilities for multi-scale and occluded targets, offering a robust solution for analyzing student behavior. SCB-DETR establishes an end-to-end framework that simplifies the detection process and consistently outperforms other deep learning methods. Employing our custom Student Classroom Behavior (SCBehavior) Dataset, SCB-DETR achieves a mean Average Precision (mAP) of 0.626, which is a 1.5% improvement over the baseline model's mAP and a 6% increase in AP50. These results demonstrate SCB-DETR's superior performance in handling the uneven distribution of student behaviors and ensuring precise detection in dynamic classroom environments.

Title: Masked Generative Priors Improve World Models Sequence Modelling Capabilities

• Authors: Cristian Meo, Mircea Lica, Zarif Ikram, Akihiro Nakano, Vedant Shah, Aniket Rajiv Didolkar, Dianbo Liu, Anirudh Goyal, Justin Dauwels
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07836
• Pdf URL: https://arxiv.org/pdf/2410.07836
• Copy Paste: [[2410.07836]] Masked Generative Priors Improve World Models Sequence Modelling Capabilities(https://arxiv.org/abs/2410.07836)
• Keywords: transformer, generative
• Abstract: Deep Reinforcement Learning (RL) has become the leading approach for creating artificial agents in complex environments. Model-based approaches, which are RL methods with world models that predict environment dynamics, are among the most promising directions for improving data efficiency, forming a critical step toward bridging the gap between research and real-world deployment. In particular, world models enhance sample efficiency by learning in imagination, which involves training a generative sequence model of the environment in a self-supervised manner. Recently, Masked Generative Modelling has emerged as a more efficient and superior inductive bias for modelling and generating token sequences. Building on the Efficient Stochastic Transformer-based World Models (STORM) architecture, we replace the traditional MLP prior with a Masked Generative Prior (e.g., MaskGIT Prior) and introduce GIT-STORM. We evaluate our model on two downstream tasks: reinforcement learning and video prediction. GIT-STORM demonstrates substantial performance gains in RL tasks on the Atari 100k benchmark. Moreover, we apply Transformer-based World Models to continuous action environments for the first time, addressing a significant gap in prior research. To achieve this, we employ a state mixer function that integrates latent state representations with actions, enabling our model to handle continuous control tasks. We validate this approach through qualitative and quantitative analyses on the DeepMind Control Suite, showcasing the effectiveness of Transformer-based World Models in this new domain. Our results highlight the versatility and efficacy of the MaskGIT dynamics prior, paving the way for more accurate world models and effective RL policies.

Title: MinorityPrompt: Text to Minority Image Generation via Prompt Optimization

• Authors: Soobin Um, Jong Chul Ye
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07838
• Pdf URL: https://arxiv.org/pdf/2410.07838
• Copy Paste: [[2410.07838]] MinorityPrompt: Text to Minority Image Generation via Prompt Optimization(https://arxiv.org/abs/2410.07838)
• Keywords: diffusion
• Abstract: We investigate the generation of minority samples using pretrained text-to-image (T2I) latent diffusion models. Minority instances, in the context of T2I generation, can be defined as ones living on low-density regions of text-conditional data distributions. They are valuable for various applications of modern T2I generators, such as data augmentation and creative AI. Unfortunately, existing pretrained T2I diffusion models primarily focus on high-density regions, largely due to the influence of guided samplers (like CFG) that are essential for producing high-quality generations. To address this, we present a novel framework to counter the high-density-focus of T2I diffusion models. Specifically, we first develop an online prompt optimization framework that can encourage the emergence of desired properties during inference while preserving semantic contents of user-provided prompts. We subsequently tailor this generic prompt optimizer into a specialized solver that promotes the generation of minority features by incorporating a carefully-crafted likelihood objective. Our comprehensive experiments, conducted across various types of T2I models, demonstrate that our approach significantly enhances the capability to produce high-quality minority instances compared to existing samplers.

Title: Enhancing Language Model Reasoning via Weighted Reasoning in Self-Consistency

• Authors: Tim Knappe, Ryan Li, Ayush Chauhan, Kaylee Chhua, Kevin Zhu, Sean O'Brien
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07839
• Pdf URL: https://arxiv.org/pdf/2410.07839
• Copy Paste: [[2410.07839]] Enhancing Language Model Reasoning via Weighted Reasoning in Self-Consistency(https://arxiv.org/abs/2410.07839)
• Keywords: robust, large language model
• Abstract: While large language models (LLMs) have rapidly improved their performance on a broad number of tasks, they still often fall short on reasoning tasks. As LLMs become more integrated in diverse real-world tasks, advancing their reasoning capabilities is crucial to their effectiveness in nuanced, complex problems. Wang et al's self-consistency framework reveals that sampling multiple rationales before taking a majority vote reliably improves model performance across various closed-answer reasoning tasks. Standard methods based on this framework aggregate the final decisions of these rationales but fail to utilize the detailed step-by-step reasoning paths applied by these paths. Our work enhances this approach by incorporating and analyzing both the reasoning paths of these rationales in addition to their final decisions before taking a majority vote. These methods not only improve the reliability of reasoning paths but also cause more robust performance on complex reasoning tasks.

Title: Protect Before Generate: Error Correcting Codes within Discrete Deep Generative Models

• Authors: María Martínez-García, Grace Villacrés, David Mitchell, Pablo M. Olmos
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07840
• Pdf URL: https://arxiv.org/pdf/2410.07840
• Copy Paste: [[2410.07840]] Protect Before Generate: Error Correcting Codes within Discrete Deep Generative Models(https://arxiv.org/abs/2410.07840)
• Keywords: protect, robust, generative
• Abstract: Despite significant advancements in deep probabilistic models, learning low-dimensional discrete latent representations remains a challenging task. In this paper, we introduce a novel method that enhances variational inference in discrete latent variable models by leveraging Error Correcting Codes (ECCs) to introduce redundancy in the latent representations. This redundancy is then exploited by the variational posterior to yield more accurate estimates, thereby narrowing the variational gap. Inspired by ECCs commonly used in digital communications and data storage, we demonstrate proof-of-concept using a Discrete Variational Autoencoder (DVAE) with binary latent variables and block repetition codes. We further extend this idea to a hierarchical structure based on polar codes, where certain latent bits are more robustly protected. Our method improves generation quality, data reconstruction, and uncertainty calibration compared to the uncoded DVAE, even when trained with tighter bounds such as the Importance Weighted Autoencoder (IWAE) objective. In particular, we demonstrate superior performance on MNIST, FMNIST, CIFAR10, and Tiny ImageNet datasets. The general approach of integrating ECCs into variational inference is compatible with existing techniques to boost variational inference, such as importance sampling or Hamiltonian Monte Carlo. We also outline the key properties ECCs must have to effectively enhance discrete variational inference.

Title: Scalable Representation Learning for Multimodal Tabular Transactions

• Authors: Natraj Raman, Sumitra Ganesh, Manuela Veloso
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07851
• Pdf URL: https://arxiv.org/pdf/2410.07851
• Copy Paste: [[2410.07851]] Scalable Representation Learning for Multimodal Tabular Transactions(https://arxiv.org/abs/2410.07851)
• Keywords: large language model
• Abstract: Large language models (LLMs) are primarily designed to understand unstructured text. When directly applied to structured formats such as tabular data, they may struggle to discern inherent relationships and overlook critical patterns. While tabular representation learning methods can address some of these limitations, existing efforts still face challenges with sparse high-cardinality fields, precise numerical reasoning, and column-heavy tables. Furthermore, leveraging these learned representations for downstream tasks through a language based interface is not apparent. In this paper, we present an innovative and scalable solution to these challenges. Concretely, our approach introduces a multi-tier partitioning mechanism that utilizes power-law dynamics to handle large vocabularies, an adaptive quantization mechanism to impose priors on numerical continuity, and a distinct treatment of core-columns and meta-information columns. To facilitate instruction tuning on LLMs, we propose a parameter efficient decoder that interleaves transaction and text modalities using a series of adapter layers, thereby exploiting rich cross-task knowledge. We validate the efficacy of our solution on a large-scale dataset of synthetic payments transactions.

Title: SNN-PAR: Energy Efficient Pedestrian Attribute Recognition via Spiking Neural Networks

• Authors: Haiyang Wang, Qian Zhu, Mowen She, Yabo Li, Haoyu Song, Minghe Xu, Xiao Wang
• Subjects: cs.CV, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2410.07857
• Pdf URL: https://arxiv.org/pdf/2410.07857
• Copy Paste: [[2410.07857]] SNN-PAR: Energy Efficient Pedestrian Attribute Recognition via Spiking Neural Networks(https://arxiv.org/abs/2410.07857)
• Keywords: extraction, transformer
• Abstract: Artificial neural network based Pedestrian Attribute Recognition (PAR) has been widely studied in recent years, despite many progresses, however, the energy consumption is still high. To address this issue, in this paper, we propose a Spiking Neural Network (SNN) based framework for energy-efficient attribute recognition. Specifically, we first adopt a spiking tokenizer module to transform the given pedestrian image into spiking feature representations. Then, the output will be fed into the spiking Transformer backbone networks for energy-efficient feature extraction. We feed the enhanced spiking features into a set of feed-forward networks for pedestrian attribute recognition. In addition to the widely used binary cross-entropy loss function, we also exploit knowledge distillation from the artificial neural network to the spiking Transformer network for more accurate attribute recognition. Extensive experiments on three widely used PAR benchmark datasets fully validated the effectiveness of our proposed SNN-PAR framework. The source code of this paper is released on \url{this https URL}.

Title: BA-Net: Bridge Attention in Deep Neural Networks

• Authors: Ronghui Zhang, Runzong Zou, Yue Zhao, Zirui Zhang, Junzhou Chen, Yue Cao, Chuan Hu, Houbing Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07860
• Pdf URL: https://arxiv.org/pdf/2410.07860
• Copy Paste: [[2410.07860]] BA-Net: Bridge Attention in Deep Neural Networks(https://arxiv.org/abs/2410.07860)
• Keywords: transformer
• Abstract: Attention mechanisms, particularly channel attention, have become highly influential in numerous computer vision tasks. Despite their effectiveness, many existing methods primarily focus on optimizing performance through complex attention modules applied at individual convolutional layers, often overlooking the synergistic interactions that can occur across multiple layers. In response to this gap, we introduce bridge attention, a novel approach designed to facilitate more effective integration and information flow between different convolutional layers. Our work extends the original bridge attention model (BAv1) by introducing an adaptive selection operator, which reduces information redundancy and optimizes the overall information exchange. This enhancement results in the development of BAv2, which achieves substantial performance improvements in the ImageNet classification task, obtaining Top-1 accuracies of 80.49% and 81.75% when using ResNet50 and ResNet101 as backbone networks, respectively. These results surpass the retrained baselines by 1.61% and 0.77%, respectively. Furthermore, BAv2 outperforms other existing channel attention techniques, such as the classical SENet101, exceeding its retrained performance by 0.52% Additionally, integrating BAv2 into advanced convolutional networks and vision transformers has led to significant gains in performance across a wide range of computer vision tasks, underscoring its broad applicability.

Title: Benchmarking Agentic Workflow Generation

• Authors: Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2410.07869
• Pdf URL: https://arxiv.org/pdf/2410.07869
• Copy Paste: [[2410.07869]] Benchmarking Agentic Workflow Generation(https://arxiv.org/abs/2410.07869)
• Keywords: large language model
• Abstract: Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is a crucial step in this process. Existing workflow evaluation frameworks either focus solely on holistic performance or suffer from limitations such as restricted scenario coverage, simplistic workflow structures, and lax evaluation standards. To this end, we introduce WorFBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. Additionally, we present WorFEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms to accurately quantify the LLM agent's workflow generation capabilities. Through comprehensive evaluations across different types of LLMs, we discover distinct gaps between the sequence planning capabilities and graph planning capabilities of LLM agents, with even GPT-4 exhibiting a gap of around 15%. We also train two open-source models and evaluate their generalization abilities on held-out tasks. Furthermore, we observe that the generated workflows can enhance downstream tasks, enabling them to achieve superior performance with less time during inference. Code and dataset will be available at this https URL.

Title: A Comprehensive Survey on Joint Resource Allocation Strategies in Federated Edge Learning

• Authors: Jingbo Zhang, Qiong Wu, Pingyi Fan, Qiang Fan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07881
• Pdf URL: https://arxiv.org/pdf/2410.07881
• Copy Paste: [[2410.07881]] A Comprehensive Survey on Joint Resource Allocation Strategies in Federated Edge Learning(https://arxiv.org/abs/2410.07881)
• Keywords: privacy, robust, federate
• Abstract: Federated Edge Learning (FEL), an emerging distributed Machine Learning (ML) paradigm, enables model training in a distributed environment while ensuring user privacy by using physical separation for each user data. However, with the development of complex application scenarios such as the Internet of Things (IoT) and Smart Earth, the conventional resource allocation schemes can no longer effectively support these growing computational and communication demands. Therefore, joint resource optimization may be the key solution to the scaling problem. This paper simultaneously addresses the multifaceted challenges of computation and communication, with the growing multiple resource demands. We systematically review the joint allocation strategies for different resources (computation, data, communication, and network topology) in FEL, and summarize the advantages in improving system efficiency, reducing latency, enhancing resource utilization and enhancing robustness. In addition, we present the potential ability of joint optimization to enhance privacy preservation by reducing communication requirements, indirectly. This work not only provides theoretical support for resource management in federated learning (FL) systems, but also provides ideas for potential optimal deployment in multiple real-world scenarios. By thoroughly discussing the current challenges and future research directions, it also provides some important insights into multi-resource optimization in complex application environments.

Title: Generated Bias: Auditing Internal Bias Dynamics of Text-To-Image Generative Models

• Authors: Abhishek Mandal, Susan Leavy, Suzanne Little
• Subjects: cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.07884
• Pdf URL: https://arxiv.org/pdf/2410.07884
• Copy Paste: [[2410.07884]] Generated Bias: Auditing Internal Bias Dynamics of Text-To-Image Generative Models(https://arxiv.org/abs/2410.07884)
• Keywords: diffusion, generative
• Abstract: Text-To-Image (TTI) Diffusion Models such as DALL-E and Stable Diffusion are capable of generating images from text prompts. However, they have been shown to perpetuate gender stereotypes. These models process data internally in multiple stages and employ several constituent models, often trained separately. In this paper, we propose two novel metrics to measure bias internally in these multistage multimodal models. Diffusion Bias was developed to detect and measures bias introduced by the diffusion stage of the models. Bias Amplification measures amplification of bias during the text-to-image conversion process. Our experiments reveal that TTI models amplify gender bias, the diffusion process itself contributes to bias and that Stable Diffusion v2 is more prone to gender bias than DALL-E 2.

Title: Deepfake detection in videos with multiple faces using geometric-fakeness features

• Authors: Kirill Vyshegorodtsev, Dmitry Kudiyarov, Alexander Balashov, Alexander Kuzmin
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.07888
• Pdf URL: https://arxiv.org/pdf/2410.07888
• Copy Paste: [[2410.07888]] Deepfake detection in videos with multiple faces using geometric-fakeness features(https://arxiv.org/abs/2410.07888)
• Keywords: security, privacy, biometric
• Abstract: Due to the development of facial manipulation techniques in recent years deepfake detection in video stream became an important problem for face biometrics, brand monitoring or online video conferencing solutions. In case of a biometric authentication, if you replace a real datastream with a deepfake, you can bypass a liveness detection system. Using a deepfake in a video conference, you can penetrate into a private meeting. Deepfakes of victims or public figures can also be used by fraudsters for blackmailing, extorsion and financial fraud. Therefore, the task of detecting deepfakes is relevant to ensuring privacy and security. In existing approaches to a deepfake detection their performance deteriorates when multiple faces are present in a video simultaneously or when there are other objects erroneously classified as faces. In our research we propose to use geometric-fakeness features (GFF) that characterize a dynamic degree of a face presence in a video and its per-frame deepfake scores. To analyze temporal inconsistencies in GFFs between the frames we train a complex deep learning model that outputs a final deepfake prediction. We employ our approach to analyze videos with multiple faces that are simultaneously present in a video. Such videos often occur in practice e.g., in an online video conference. In this case, real faces appearing in a frame together with a deepfake face will significantly affect a deepfake detection and our approach allows to counter this problem. Through extensive experiments we demonstrate that our approach outperforms current state-of-the-art methods on popular benchmark datasets such as FaceForensics++, DFDC, Celeb-DF and WildDeepFake. The proposed approach remains accurate when trained to detect multiple different deepfake generation techniques.

Title: Ormer: A Manipulation-resistant and Gas-efficient Blockchain Pricing Oracle for DeFi

• Authors: Dongbin Bai, Jiannong Cao, Yinfeng Cao, Long Wen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.07893
• Pdf URL: https://arxiv.org/pdf/2410.07893
• Copy Paste: [[2410.07893]] Ormer: A Manipulation-resistant and Gas-efficient Blockchain Pricing Oracle for DeFi(https://arxiv.org/abs/2410.07893)
• Keywords: attack
• Abstract: Blockchain oracle is a critical third-party web service for Decentralized Finance (DeFi) protocols. Oracles retrieve external information such as token prices from exchanges and feed them as trusted data sources into smart contracts, enabling core DeFi applications such as loaning protocols. Currently, arithmetic mean based time-weighted average price (TWAP) oracles are widely used in DeFi by averaging external price data with fixed time frame, which is considered reliable and gas-efficient for protocol execution. However, recent research shows that TWAP price feeds are vulnerable to price manipulation attack even with long time frame setting, which would further introduce long time delays and price errors hindering the service quality of DeFi applications. To address this issue, we propose a novel on-chain gas-efficient pricing algorithm (Ormer) that heuristically estimates the median of the current streaming asset price feed based on a piecewise-parabolic formula, while the time delay is suppressed by fusing estimations with different observation window size. Our evaluation based on Ethereum WETH/USDT swapping pair price feed shows that Ormer reduces the mean absolute price error by 15.3% and the time delay by 49.3% compared to TWAP. For gas efficiency, an optimized smart contract design and constant storage requirement regardless of the number of price observations is developed for Ormer.

Title: CL3: A Collaborative Learning Framework for the Medical Data Ensuring Data Privacy in the Hyperconnected Environment

• Authors: Mohamamd Zavid Parvez, Rafiqul Islam, Md Zahidul Islam
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07900
• Pdf URL: https://arxiv.org/pdf/2410.07900
• Copy Paste: [[2410.07900]] CL3: A Collaborative Learning Framework for the Medical Data Ensuring Data Privacy in the Hyperconnected Environment(https://arxiv.org/abs/2410.07900)
• Keywords: secure, privacy, federate
• Abstract: In a hyperconnected environment, medical institutions are particularly concerned with data privacy when sharing and transmitting sensitive patient information due to the risk of data breaches, where malicious actors could intercept sensitive information. A collaborative learning framework, including transfer, federated, and incremental learning, can generate efficient, secure, and scalable models while requiring less computation, maintaining patient data privacy, and ensuring an up-to-date model. This study aims to address the detection of COVID-19 using chest X-ray images through a proposed collaborative learning framework called CL3. Initially, transfer learning is employed, leveraging knowledge from a pre-trained model as the starting global model. Local models from different medical institutes are then integrated, and a new global model is constructed to adapt to any data drift observed in the local models. Additionally, incremental learning is considered, allowing continuous adaptation to new medical data without forgetting previously learned information. Experimental results demonstrate that the CL3 framework achieved a global accuracy of 89.99\% when using Xception with a batch size of 16 after being trained for six federated communication rounds.

Title: Understanding Spatio-Temporal Relations in Human-Object Interaction using Pyramid Graph Convolutional Network

• Authors: Hao Xing, Darius Burschka
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.07912
• Pdf URL: https://arxiv.org/pdf/2410.07912
• Copy Paste: [[2410.07912]] Understanding Spatio-Temporal Relations in Human-Object Interaction using Pyramid Graph Convolutional Network(https://arxiv.org/abs/2410.07912)
• Keywords: segmentation
• Abstract: Human activities recognition is an important task for an intelligent robot, especially in the field of human-robot collaboration, it requires not only the label of sub-activities but also the temporal structure of the activity. In order to automatically recognize both the label and the temporal structure in sequence of human-object interaction, we propose a novel Pyramid Graph Convolutional Network (PGCN), which employs a pyramidal encoder-decoder architecture consisting of an attention based graph convolution network and a temporal pyramid pooling module for downsampling and upsampling interaction sequence on the temporal axis, respectively. The system represents the 2D or 3D spatial relation of human and objects from the detection results in video data as a graph. To learn the human-object relations, a new attention graph convolutional network is trained to extract condensed information from the graph representation. To segment action into sub-actions, a novel temporal pyramid pooling module is proposed, which upsamples compressed features back to the original time scale and classifies actions per frame. We explore various attention layers, namely spatial attention, temporal attention and channel attention, and combine different upsampling decoders to test the performance on action recognition and segmentation. We evaluate our model on two challenging datasets in the field of human-object interaction recognition, i.e. Bimanual Actions and IKEA Assembly datasets. We demonstrate that our classifier significantly improves both framewise action recognition and segmentation, e.g., F1 micro and F1@50 scores on Bimanual Actions dataset are improved by $4.3\%$ and $8.5\%$ respectively.

Title: Robustness Auditing for Linear Regression: To Singularity and Beyond

• Authors: Ittai Rubinstein, Samuel B. Hopkins
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07916
• Pdf URL: https://arxiv.org/pdf/2410.07916
• Copy Paste: [[2410.07916]] Robustness Auditing for Linear Regression: To Singularity and Beyond(https://arxiv.org/abs/2410.07916)
• Keywords: robust
• Abstract: It has recently been discovered that the conclusions of many highly influential econometrics studies can be overturned by removing a very small fraction of their samples (often less than $0.5\%$). These conclusions are typically based on the results of one or more Ordinary Least Squares (OLS) regressions, raising the question: given a dataset, can we certify the robustness of an OLS fit on this dataset to the removal of a given number of samples? Brute-force techniques quickly break down even on small datasets. Existing approaches which go beyond brute force either can only find candidate small subsets to remove (but cannot certify their non-existence) [BGM20, KZC21], are computationally intractable beyond low dimensional settings [MR22], or require very strong assumptions on the data distribution and too many samples to give reasonable bounds in practice [BP21, FH23]. We present an efficient algorithm for certifying the robustness of linear regressions to removals of samples. We implement our algorithm and run it on several landmark econometrics datasets with hundreds of dimensions and tens of thousands of samples, giving the first non-trivial certificates of robustness to sample removal for datasets of dimension $4$ or greater. We prove that under distributional assumptions on a dataset, the bounds produced by our algorithm are tight up to a $1 + o(1)$ multiplicative factor.

Title: InstructBioMol: Advancing Biomolecule Understanding and Design Following Human Instructions

• Authors: Xiang Zhuang, Keyan Ding, Tianwen Lyu, Yinuo Jiang, Xiaotong Li, Zhuoyi Xiang, Zeyuan Wang, Ming Qin, Kehua Feng, Jike Wang, Qiang Zhang, Huajun Chen
• Subjects: cs.CL, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.07919
• Pdf URL: https://arxiv.org/pdf/2410.07919
• Copy Paste: [[2410.07919]] InstructBioMol: Advancing Biomolecule Understanding and Design Following Human Instructions(https://arxiv.org/abs/2410.07919)
• Keywords: large language model
• Abstract: Understanding and designing biomolecules, such as proteins and small molecules, is central to advancing drug discovery, synthetic biology, and enzyme engineering. Recent breakthroughs in Artificial Intelligence (AI) have revolutionized biomolecular research, achieving remarkable accuracy in biomolecular prediction and design. However, a critical gap remains between AI's computational power and researchers' intuition, using natural language to align molecular complexity with human intentions. Large Language Models (LLMs) have shown potential to interpret human intentions, yet their application to biomolecular research remains nascent due to challenges including specialized knowledge requirements, multimodal data integration, and semantic alignment between natural language and biomolecules. To address these limitations, we present InstructBioMol, a novel LLM designed to bridge natural language and biomolecules through a comprehensive any-to-any alignment of natural language, molecules, and proteins. This model can integrate multimodal biomolecules as input, and enable researchers to articulate design goals in natural language, providing biomolecular outputs that meet precise biological needs. Experimental results demonstrate InstructBioMol can understand and design biomolecules following human instructions. Notably, it can generate drug molecules with a 10% improvement in binding affinity and design enzymes that achieve an ESP Score of 70.4, making it the only method to surpass the enzyme-substrate interaction threshold of 60.0 recommended by the ESP developer. This highlights its potential to transform real-world biomolecular research.

Title: Efficient Reinforcement Learning with Large Language Model Priors

• Authors: Xue Yan, Yan Song, Xidong Feng, Mengyue Yang, Haifeng Zhang, Haitham Bou Ammar, Jun Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07927
• Pdf URL: https://arxiv.org/pdf/2410.07927
• Copy Paste: [[2410.07927]] Efficient Reinforcement Learning with Large Language Model Priors(https://arxiv.org/abs/2410.07927)
• Keywords: large language model
• Abstract: In sequential decision-making (SDM) tasks, methods like reinforcement learning (RL) and heuristic search have made notable advances in specific cases. However, they often require extensive exploration and face challenges in generalizing across diverse environments due to their limited grasp of the underlying decision dynamics. In contrast, large language models (LLMs) have recently emerged as powerful general-purpose tools, due to their capacity to maintain vast amounts of domain-specific knowledge. To harness this rich prior knowledge for efficiently solving complex SDM tasks, we propose treating LLMs as prior action distributions and integrating them into RL frameworks through Bayesian inference methods, making use of variational inference and direct posterior sampling. The proposed approaches facilitate the seamless incorporation of fixed LLM priors into both policy-based and value-based RL frameworks. Our experiments show that incorporating LLM-based action priors significantly reduces exploration and optimization complexity, substantially improving sample efficiency compared to traditional RL techniques, e.g., using LLM priors decreases the number of required samples by over 90% in offline learning scenarios.

Title: Offline Hierarchical Reinforcement Learning via Inverse Optimization

• Authors: Carolin Schmidt, Daniele Gammelli, James Harrison, Marco Pavone, Filipe Rodrigues
• Subjects: cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2410.07933
• Pdf URL: https://arxiv.org/pdf/2410.07933
• Copy Paste: [[2410.07933]] Offline Hierarchical Reinforcement Learning via Inverse Optimization(https://arxiv.org/abs/2410.07933)
• Keywords: robust
• Abstract: Hierarchical policies enable strong performance in many sequential decision-making problems, such as those with high-dimensional action spaces, those requiring long-horizon planning, and settings with sparse rewards. However, learning hierarchical policies from static offline datasets presents a significant challenge. Crucially, actions taken by higher-level policies may not be directly observable within hierarchical controllers, and the offline dataset might have been generated using a different policy structure, hindering the use of standard offline learning algorithms. In this work, we propose OHIO: a framework for offline reinforcement learning (RL) of hierarchical policies. Our framework leverages knowledge of the policy structure to solve the inverse problem, recovering the unobservable high-level actions that likely generated the observed data under our hierarchical policy. This approach constructs a dataset suitable for off-the-shelf offline training. We demonstrate our framework on robotic and network optimization problems and show that it substantially outperforms end-to-end RL methods and improves robustness. We investigate a variety of instantiations of our framework, both in direct deployment of policies trained offline and when online fine-tuning is performed.

Title: Disease Entity Recognition and Normalization is Improved with Large Language Model Derived Synthetic Normalized Mentions

• Authors: Kuleen Sasse, Shinjitha Vadlakonda, Richard E. Kennedy, John D. Osborne
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07951
• Pdf URL: https://arxiv.org/pdf/2410.07951
• Copy Paste: [[2410.07951]] Disease Entity Recognition and Normalization is Improved with Large Language Model Derived Synthetic Normalized Mentions(https://arxiv.org/abs/2410.07951)
• Keywords: extraction, large language model
• Abstract: Background: Machine learning methods for clinical named entity recognition and entity normalization systems can utilize both labeled corpora and Knowledge Graphs (KGs) for learning. However, infrequently occurring concepts may have few mentions in training corpora and lack detailed descriptions or synonyms, even in large KGs. For Disease Entity Recognition (DER) and Disease Entity Normalization (DEN), this can result in fewer high quality training examples relative to the number of known diseases. Large Language Model (LLM) generation of synthetic training examples could improve performance in these information extraction tasks. Methods: We fine-tuned a LLaMa-2 13B Chat LLM to generate a synthetic corpus containing normalized mentions of concepts from the Unified Medical Language System (UMLS) Disease Semantic Group. We measured overall and Out of Distribution (OOD) performance for DER and DEN, with and without synthetic data augmentation. We evaluated performance on 3 different disease corpora using 4 different data augmentation strategies, assessed using BioBERT for DER and SapBERT and KrissBERT for DEN. Results: Our synthetic data yielded a substantial improvement for DEN, in all 3 training corpora the top 1 accuracy of both SapBERT and KrissBERT improved by 3-9 points in overall performance and by 20-55 points in OOD data. A small improvement (1-2 points) was also seen for DER in overall performance, but only one dataset showed OOD improvement. Conclusion: LLM generation of normalized disease mentions can improve DEN relative to normalization approaches that do not utilize LLMs to augment data with synthetic mentions. Ablation studies indicate that performance gains for DEN were only partially attributable to improvements in OOD performance. The same approach has only a limited ability to improve DER. We make our software and dataset publicly available.

Title: Iterative Optimization Annotation Pipeline and ALSS-YOLO-Seg for Efficient Banana Plantation Segmentation in UAV Imagery

• Authors: Ang He, Ximei Wu, Xing Xu, Jing Chen, Xiaobin Guo, Sheng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07955
• Pdf URL: https://arxiv.org/pdf/2410.07955
• Copy Paste: [[2410.07955]] Iterative Optimization Annotation Pipeline and ALSS-YOLO-Seg for Efficient Banana Plantation Segmentation in UAV Imagery(https://arxiv.org/abs/2410.07955)
• Keywords: extraction, segmentation
• Abstract: Precise segmentation of Unmanned Aerial Vehicle (UAV)-captured images plays a vital role in tasks such as crop yield estimation and plant health assessment in banana plantations. By identifying and classifying planted areas, crop area can be calculated, which is indispensable for accurate yield predictions. However, segmenting banana plantation scenes requires a substantial amount of annotated data, and manual labeling of these images is both time-consuming and labor-intensive, limiting the development of large-scale datasets. Furthermore, challenges such as changing target sizes, complex ground backgrounds, limited computational resources, and correct identification of crop categories make segmentation even more difficult. To address these issues, we proposed a comprehensive solution. Firstly, we designed an iterative optimization annotation pipeline leveraging SAM2's zero-shot capabilities to generate high-quality segmentation annotations, thereby reducing the cost and time associated with data annotation significantly. Secondly, we developed ALSS-YOLO-Seg, an efficient lightweight segmentation model optimized for UAV imagery. The model's backbone includes an Adaptive Lightweight Channel Splitting and Shuffling (ALSS) module to improve information exchange between channels and optimize feature extraction, aiding accurate crop identification. Additionally, a Multi-Scale Channel Attention (MSCA) module combines multi-scale feature extraction with channel attention to tackle challenges of varying target sizes and complex ground backgrounds.

Title: COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act

• Authors: Philipp Guldimann, Alexander Spiridonov, Robin Staab, Nikola Jovanović, Mark Vero, Velko Vechev, Anna Gueorguieva, Mislav Balunović, Nikola Konstantinov, Pavol Bielik, Petar Tsankov, Martin Vechev
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.07959
• Pdf URL: https://arxiv.org/pdf/2410.07959
• Copy Paste: [[2410.07959]] COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act(https://arxiv.org/abs/2410.07959)
• Keywords: robust, fair, large language model
• Abstract: The EU's Artificial Intelligence Act (AI Act) is a significant step towards responsible AI development, but lacks clear technical interpretation, making it difficult to assess models' compliance. This work presents COMPL-AI, a comprehensive framework consisting of (i) the first technical interpretation of the EU AI Act, translating its broad regulatory requirements into measurable technical requirements, with the focus on large language models (LLMs), and (ii) an open-source Act-centered benchmarking suite, based on thorough surveying and implementation of state-of-the-art LLM benchmarks. By evaluating 12 prominent LLMs in the context of COMPL-AI, we reveal shortcomings in existing models and benchmarks, particularly in areas like robustness, safety, diversity, and fairness. This work highlights the need for a shift in focus towards these aspects, encouraging balanced development of LLMs and more comprehensive regulation-aligned benchmarks. Simultaneously, COMPL-AI for the first time demonstrates the possibilities and difficulties of bringing the Act's obligations to a more concrete, technical level. As such, our work can serve as a useful first step towards having actionable recommendations for model providers, and contributes to ongoing efforts of the EU to enable application of the Act, such as the drafting of the GPAI Code of Practice.

Title: Neural Reasoning Networks: Efficient Interpretable Neural Networks With Automatic Textual Explanations

• Authors: Stephen Carrow, Kyle Harper Erwin, Olga Vilenskaia, Parikshit Ram, Tim Klinger, Naweed Aghmad Khan, Ndivhuwo Makondo, Alexander Gray
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07966
• Pdf URL: https://arxiv.org/pdf/2410.07966
• Copy Paste: [[2410.07966]] Neural Reasoning Networks: Efficient Interpretable Neural Networks With Automatic Textual Explanations(https://arxiv.org/abs/2410.07966)
• Keywords: fair, interpretability
• Abstract: Recent advances in machine learning have led to a surge in adoption of neural networks for various tasks, but lack of interpretability remains an issue for many others in which an understanding of the features influencing the prediction is necessary to ensure fairness, safety, and legal compliance. In this paper we consider one class of such tasks, tabular dataset classification, and propose a novel neuro-symbolic architecture, Neural Reasoning Networks (NRN), that is scalable and generates logically sound textual explanations for its predictions. NRNs are connected layers of logical neurons which implement a form of real valued logic. A training algorithm (R-NRN) learns the weights of the network as usual using gradient descent optimization with backprop, but also learns the network structure itself using a bandit-based optimization. Both are implemented in an extension to PyTorch (this https URL) that takes full advantage of GPU scaling and batched training. Evaluation on a diverse set of 22 open-source datasets for tabular classification demonstrates performance (measured by ROC AUC) which improves over multi-layer perceptron (MLP) and is statistically similar to other state-of-the-art approaches such as Random Forest, XGBoost and Gradient Boosted Trees, while offering 43% faster training and a more than 2 orders of magnitude reduction in the number of parameters required, on average. Furthermore, R-NRN explanations are shorter than the compared approaches while producing more accurate feature importance scores.

Title: MolMix: A Simple Yet Effective Baseline for Multimodal Molecular Representation Learning

• Authors: Andrei Manolache, Dragos Tantaru, Mathias Niepert
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.07981
• Pdf URL: https://arxiv.org/pdf/2410.07981
• Copy Paste: [[2410.07981]] MolMix: A Simple Yet Effective Baseline for Multimodal Molecular Representation Learning(https://arxiv.org/abs/2410.07981)
• Keywords: transformer
• Abstract: In this work, we propose a simple transformer-based baseline for multimodal molecular representation learning, integrating three distinct modalities: SMILES strings, 2D graph representations, and 3D conformers of molecules. A key aspect of our approach is the aggregation of 3D conformers, allowing the model to account for the fact that molecules can adopt multiple conformations-an important factor for accurate molecular representation. The tokens for each modality are extracted using modality-specific encoders: a transformer for SMILES strings, a message-passing neural network for 2D graphs, and an equivariant neural network for 3D conformers. The flexibility and modularity of this framework enable easy adaptation and replacement of these encoders, making the model highly versatile for different molecular tasks. The extracted tokens are then combined into a unified multimodal sequence, which is processed by a downstream transformer for prediction tasks. To efficiently scale our model for large multimodal datasets, we utilize Flash Attention 2 and bfloat16 precision. Despite its simplicity, our approach achieves state-of-the-art results across multiple datasets, demonstrating its effectiveness as a strong baseline for multimodal molecular representation learning.

Title: Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models

• Authors: Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, Baobao Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.07985
• Pdf URL: https://arxiv.org/pdf/2410.07985
• Copy Paste: [[2410.07985]] Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models(https://arxiv.org/abs/2410.07985)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have led to significant breakthroughs in mathematical reasoning capabilities. However, existing benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g., OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for truly challenging these models. To bridge this gap, we propose a comprehensive and challenging benchmark specifically designed to assess LLMs' mathematical reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks, our dataset focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels, enabling a holistic assessment of model performance in Olympiad-mathematical reasoning. Furthermore, we conducted an in-depth analysis based on this benchmark. Our experimental results show that even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle with highly challenging Olympiad-level problems, with 60.54% and 52.55% accuracy, highlighting significant challenges in Olympiad-level mathematical reasoning.

Title: LADIMO: Face Morph Generation through Biometric Template Inversion with Latent Diffusion

• Authors: Marcel Grimmer, Christoph Busch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07988
• Pdf URL: https://arxiv.org/pdf/2410.07988
• Copy Paste: [[2410.07988]] LADIMO: Face Morph Generation through Biometric Template Inversion with Latent Diffusion(https://arxiv.org/abs/2410.07988)
• Keywords: security, attack, biometric, diffusion
• Abstract: Face morphing attacks pose a severe security threat to face recognition systems, enabling the morphed face image to be verified against multiple identities. To detect such manipulated images, the development of new face morphing methods becomes essential to increase the diversity of training datasets used for face morph detection. In this study, we present a representation-level face morphing approach, namely LADIMO, that performs morphing on two face recognition embeddings. Specifically, we train a Latent Diffusion Model to invert a biometric template - thus reconstructing the face image from an FRS latent representation. Our subsequent vulnerability analysis demonstrates the high morph attack potential in comparison to MIPGAN-II, an established GAN-based face morphing approach. Finally, we exploit the stochastic LADMIO model design in combination with our identity conditioning mechanism to create unlimited morphing attacks from a single face morph image pair. We show that each face morph variant has an individual attack success rate, enabling us to maximize the morph attack potential by applying a simple re-sampling strategy. Code and pre-trained models available here: this https URL

Title: RegionGrasp: A Novel Task for Contact Region Controllable Hand Grasp Generation

• Authors: Yilin Wang, Chuan Guo, Li Cheng, Hai Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.07995
• Pdf URL: https://arxiv.org/pdf/2410.07995
• Copy Paste: [[2410.07995]] RegionGrasp: A Novel Task for Contact Region Controllable Hand Grasp Generation(https://arxiv.org/abs/2410.07995)
• Keywords: transformer
• Abstract: Can machine automatically generate multiple distinct and natural hand grasps, given specific contact region of an object in 3D? This motivates us to consider a novel task of \textit{Region Controllable Hand Grasp Generation (RegionGrasp)}, as follows: given as input a 3D object, together with its specific surface area selected as the intended contact region, to generate a diverse set of plausible hand grasps of the object, where the thumb finger tip touches the object surface on the contact region. To address this task, RegionGrasp-CVAE is proposed, which consists of two main parts. First, to enable contact region-awareness, we propose ConditionNet as the condition encoder that includes in it a transformer-backboned object encoder, O-Enc; a pretraining strategy is adopted by O-Enc, where the point patches of object surface are randomly masked off and subsequently restored, to further capture surface geometric information of the object. Second, to realize interaction awareness, HOINet is introduced to encode hand-object interaction features by entangling high-level hand features with embedded object features through geometric-aware multi-head cross attention. Empirical evaluations demonstrate the effectiveness of our approach qualitatively and quantitatively where it is shown to compare favorably with respect to the state of the art methods.

Title: More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing

• Authors: Sagi Shaier, Francisco Pereira, Katharina von der Wense, Lawrence E Hunter, Matt Jones
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08003
• Pdf URL: https://arxiv.org/pdf/2410.08003
• Copy Paste: [[2410.08003]] More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing(https://arxiv.org/abs/2410.08003)
• Keywords: robust
• Abstract: The evolution of biological neural systems has led to both modularity and sparse coding, which enables efficiency in energy usage, and robustness across the diversity of tasks in the lifespan. In contrast, standard neural networks rely on dense, non-specialized architectures, where all model parameters are simultaneously updated to learn multiple tasks, leading to representation interference. Current sparse neural network approaches aim to alleviate this issue, but are often hindered by limitations such as 1) trainable gating functions that cause representation collapse; 2) non-overlapping experts that result in redundant computation and slow learning; and 3) reliance on explicit input or task IDs that impose significant constraints on flexibility and scalability. In this paper we propose Conditionally Overlapping Mixture of ExperTs (COMET), a general deep learning method that addresses these challenges by inducing a modular, sparse architecture with an exponential number of overlapping experts. COMET replaces the trainable gating function used in Sparse Mixture of Experts with a fixed, biologically inspired random projection applied to individual input representations. This design causes the degree of expert overlap to depend on input similarity, so that similar inputs tend to share more parameters. This facilitates positive knowledge transfer, resulting in faster learning and improved generalization. We demonstrate the effectiveness of COMET on a range of tasks, including image classification, language modeling, and regression, using several popular deep learning architectures.

Title: Time Can Invalidate Algorithmic Recourse

• Authors: Giovanni De Toni, Stefano Teso, Bruno Lepri, Andrea Passerini
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.08007
• Pdf URL: https://arxiv.org/pdf/2410.08007
• Copy Paste: [[2410.08007]] Time Can Invalidate Algorithmic Recourse(https://arxiv.org/abs/2410.08007)
• Keywords: robust
• Abstract: Algorithmic Recourse (AR) aims to provide users with actionable steps to overturn unfavourable decisions made by machine learning predictors. However, these actions often take time to implement (e.g., getting a degree can take years), and their effects may vary as the world evolves. Thus, it is natural to ask for recourse that remains valid in a dynamic environment. In this paper, we study the robustness of algorithmic recourse over time by casting the problem through the lens of causality. We demonstrate theoretically and empirically that (even robust) causal AR methods can fail over time except in the - unlikely - case that the world is stationary. Even more critically, unless the world is fully deterministic, counterfactual AR cannot be solved optimally. To account for this, we propose a simple yet effective algorithm for temporal AR that explicitly accounts for time. Our simulations on synthetic and realistic datasets show how considering time produces more resilient solutions to potential trends in the data distribution.

Title: Study of Attacks on the HHL Quantum Algorithm

• Authors: Yizhuo Tan, Hrvoje Kukina, Jakub Szefer
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2410.08010
• Pdf URL: https://arxiv.org/pdf/2410.08010
• Copy Paste: [[2410.08010]] Study of Attacks on the HHL Quantum Algorithm(https://arxiv.org/abs/2410.08010)
• Keywords: security, defense, attack
• Abstract: As the quantum research community continues to grow and new algorithms are designed, developed, and implemented, it is crucial to start thinking about security aspects and potential threats that could result in misuse of the algorithms, or jeopardize the information processed with these quantum algorithms. This work focuses on exploration of two types of potential attacks that could be deployed on a cloud-based quantum computer by an attacker circuit trying to interfere with victim circuit. The two attacks, called Improper Initialization Attack (IIA) and Higher Energy Attack (HEA), are for the first time applied to a well-known and widely used quantum algorithm: HHL. The HHL algorithm is used in the field of machine learning and big data for solving systems of linear equations. This work evaluates the effect of the attacks on different qubits within the HHL algorithm: ancilla qubit, clock qubit, and b qubit. This work demonstrates that the two attacks are able to cause incorrect results, even when only one of the qubits in the victim algorithm is attacked. Having discovered the vulnerabilities, the work motivates the need for future work to develop defense strategies for each of these attack scenarios.

Title: LLM Cascade with Multi-Objective Optimal Consideration

• Authors: Kai Zhang, Liqian Peng, Congchao Wang, Alec Go, Xiaozhong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08014
• Pdf URL: https://arxiv.org/pdf/2410.08014
• Copy Paste: [[2410.08014]] LLM Cascade with Multi-Objective Optimal Consideration(https://arxiv.org/abs/2410.08014)
• Keywords: privacy, large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities in understanding and generating natural language. However, their high deployment costs often pose a barrier to practical applications, especially. Cascading local and server models offers a promising solution to this challenge. While existing studies on LLM cascades have primarily focused on the performance-cost trade-off, real-world scenarios often involve more complex requirements. This paper introduces a novel LLM Cascade strategy with Multi-Objective Optimization, enabling LLM cascades to consider additional objectives (e.g., privacy) and better align with the specific demands of real-world applications while maintaining their original cascading abilities. Extensive experiments on three benchmarks validate the effectiveness and superiority of our approach.

Title: Non-transferable Pruning

• Authors: Ruyi Ding, Lili Su, Aidong Adam Ding, Yunsi Fei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08015
• Pdf URL: https://arxiv.org/pdf/2410.08015
• Copy Paste: [[2410.08015]] Non-transferable Pruning(https://arxiv.org/abs/2410.08015)
• Keywords: protect
• Abstract: Pretrained Deep Neural Networks (DNNs), developed from extensive datasets to integrate multifaceted knowledge, are increasingly recognized as valuable intellectual property (IP). To safeguard these models against IP infringement, strategies for ownership verification and usage authorization have emerged. Unlike most existing IP protection strategies that concentrate on restricting direct access to the model, our study addresses an extended DNN IP issue: applicability authorization, aiming to prevent the misuse of learned knowledge, particularly in unauthorized transfer learning scenarios. We propose Non-Transferable Pruning (NTP), a novel IP protection method that leverages model pruning to control a pretrained DNN's transferability to unauthorized data domains. Selective pruning can deliberately diminish a model's suitability on unauthorized domains, even with full fine-tuning. Specifically, our framework employs the alternating direction method of multipliers (ADMM) for optimizing both the model sparsity and an innovative non-transferable learning loss, augmented with Fisher space discriminative regularization, to constrain the model's generalizability to the target dataset. We also propose a novel effective metric to measure the model non-transferability: Area Under the Sample-wise Learning Curve (SLC-AUC). This metric facilitates consideration of full fine-tuning across various sample sizes. Experimental results demonstrate that NTP significantly surpasses the state-of-the-art non-transferable learning methods, with an average SLC-AUC at $-0.54$ across diverse pairs of source and target domains, indicating that models trained with NTP do not suit for transfer learning to unauthorized target domains. The efficacy of NTP is validated in both supervised and self-supervised learning contexts, confirming its applicability in real-world scenarios.

Title: OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

• Authors: Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, Changsheng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08021
• Pdf URL: https://arxiv.org/pdf/2410.08021
• Copy Paste: [[2410.08021]] OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling(https://arxiv.org/abs/2410.08021)
• Keywords: transformer, segmentation
• Abstract: Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at this https URL.

Title: GrabDAE: An Innovative Framework for Unsupervised Domain Adaptation Utilizing Grab-Mask and Denoise Auto-Encoder

• Authors: Junzhou Chen, Xuan Wen, Ronghui Zhang, Bingtao Ren, Di Wu, Zhigang Xu, Danwei Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08023
• Pdf URL: https://arxiv.org/pdf/2410.08023
• Copy Paste: [[2410.08023]] GrabDAE: An Innovative Framework for Unsupervised Domain Adaptation Utilizing Grab-Mask and Denoise Auto-Encoder(https://arxiv.org/abs/2410.08023)
• Keywords: robust
• Abstract: Unsupervised Domain Adaptation (UDA) aims to adapt a model trained on a labeled source domain to an unlabeled target domain by addressing the domain shift. Existing Unsupervised Domain Adaptation (UDA) methods often fall short in fully leveraging contextual information from the target domain, leading to suboptimal decision boundary separation during source and target domain alignment. To address this, we introduce GrabDAE, an innovative UDA framework designed to tackle domain shift in visual classification tasks. GrabDAE incorporates two key innovations: the Grab-Mask module, which blurs background information in target domain images, enabling the model to focus on essential, domain-relevant features through contrastive learning; and the Denoising Auto-Encoder (DAE), which enhances feature alignment by reconstructing features and filtering noise, ensuring a more robust adaptation to the target domain. These components empower GrabDAE to effectively handle unlabeled target domain data, significantly improving both classification accuracy and robustness. Extensive experiments on benchmark datasets, including VisDA-2017, Office-Home, and Office31, demonstrate that GrabDAE consistently surpasses state-of-the-art UDA methods, setting new performance benchmarks. By tackling UDA's critical challenges with its novel feature masking and denoising approach, GrabDAE offers both significant theoretical and practical advancements in domain adaptation.

Title: Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modeling

• Authors: Alessio Fallani, Ramil Nugmanov, Jose Arjona-Medina, Jörg Kurt Wegner, Alexandre Tkatchenko, Kostiantyn Chernichenko
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08024
• Pdf URL: https://arxiv.org/pdf/2410.08024
• Copy Paste: [[2410.08024]] Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modeling(https://arxiv.org/abs/2410.08024)
• Keywords: transformer
• Abstract: We evaluate the impact of pretraining Graph Transformer architectures on atom-level quantum-mechanical features for the modeling of absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of drug-like compounds. We compare this pretraining strategy with two others: one based on molecular quantum properties (specifically the HOMO-LUMO gap) and one using a self-supervised atom masking technique. After fine-tuning on Therapeutic Data Commons ADMET datasets, we evaluate the performance improvement in the different models observing that models pretrained with atomic quantum mechanical properties produce in general better results. We then analyse the latent representations and observe that the supervised strategies preserve the pretraining information after finetuning and that different pretrainings produce different trends in latent expressivity across layers. Furthermore, we find that models pretrained on atomic quantum mechanical properties capture more low-frequency laplacian eigenmodes of the input graph via the attention weights and produce better representations of atomic environments within the molecule. Application of the analysis to a much larger non-public dataset for microsomal clearance illustrates generalizability of the studied indicators. In this case the performances of the models are in accordance with the representation analysis and highlight, especially for the case of masking pretraining and atom-level quantum property pretraining, how model types with similar performance on public benchmarks can have different performances on large scale pharmaceutical data.

Title: Generalization Bounds and Model Complexity for Kolmogorov-Arnold Networks

• Authors: Xianyang Zhang, Huijuan Zhou
• Subjects: cs.LG, cs.NE, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.08026
• Pdf URL: https://arxiv.org/pdf/2410.08026
• Copy Paste: [[2410.08026]] Generalization Bounds and Model Complexity for Kolmogorov-Arnold Networks(https://arxiv.org/abs/2410.08026)
• Keywords: interpretability
• Abstract: Kolmogorov-Arnold Network (KAN) is a network structure recently proposed by Liu et al. (2024) that offers improved interpretability and a more parsimonious design in many science-oriented tasks compared to multi-layer perceptrons. This work provides a rigorous theoretical analysis of KAN by establishing generalization bounds for KAN equipped with activation functions that are either represented by linear combinations of basis functions or lying in a low-rank Reproducing Kernel Hilbert Space (RKHS). In the first case, the generalization bound accommodates various choices of basis functions in forming the activation functions in each layer of KAN and is adapted to different operator norms at each layer. For a particular choice of operator norms, the bound scales with the $l_1$ norm of the coefficient matrices and the Lipschitz constants for the activation functions, and it has no dependence on combinatorial parameters (e.g., number of nodes) outside of logarithmic factors. Moreover, our result does not require the boundedness assumption on the loss function and, hence, is applicable to a general class of regression-type loss functions. In the low-rank case, the generalization bound scales polynomially with the underlying ranks as well as the Lipschitz constants of the activation functions in each layer. These bounds are empirically investigated for KANs trained with stochastic gradient descent on simulated and real data sets. The numerical results demonstrate the practical relevance of these bounds.

Title: Private Language Models via Truncated Laplacian Mechanism

• Authors: Tianhao Huang, Tao Yang, Ivan Habernal, Lijie Hu, Di Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08027
• Pdf URL: https://arxiv.org/pdf/2410.08027
• Copy Paste: [[2410.08027]] Private Language Models via Truncated Laplacian Mechanism(https://arxiv.org/abs/2410.08027)
• Keywords: privacy, attack
• Abstract: Deep learning models for NLP tasks are prone to variants of privacy attacks. To prevent privacy leakage, researchers have investigated word-level perturbations, relying on the formal guarantees of differential privacy (DP) in the embedding space. However, many existing approaches either achieve unsatisfactory performance in the high privacy regime when using the Laplacian or Gaussian mechanism, or resort to weaker relaxations of DP that are inferior to the canonical DP in terms of privacy strength. This raises the question of whether a new method for private word embedding can be designed to overcome these limitations. In this paper, we propose a novel private embedding method called the high dimensional truncated Laplacian mechanism. Specifically, we introduce a non-trivial extension of the truncated Laplacian mechanism, which was previously only investigated in one-dimensional space cases. Theoretically, we show that our method has a lower variance compared to the previous private word embedding methods. To further validate its effectiveness, we conduct comprehensive experiments on private embedding and downstream tasks using three datasets. Remarkably, even in the high privacy regime, our approach only incurs a slight decrease in utility compared to the non-private scenario.

Title: Composite Learning Units: Generalized Learning Beyond Parameter Updates to Transform LLMs into Adaptive Reasoners

• Authors: Santosh Kumar Radha, Oktay Goktas
• Subjects: cs.LG, cs.AI, cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2410.08037
• Pdf URL: https://arxiv.org/pdf/2410.08037
• Copy Paste: [[2410.08037]] Composite Learning Units: Generalized Learning Beyond Parameter Updates to Transform LLMs into Adaptive Reasoners(https://arxiv.org/abs/2410.08037)
• Keywords: large language model
• Abstract: Human learning thrives on the ability to learn from mistakes, adapt through feedback, and refine understanding-processes often missing in static machine learning models. In this work, we introduce Composite Learning Units (CLUs) designed to transform reasoners, such as Large Language Models (LLMs), into learners capable of generalized, continuous learning without conventional parameter updates while enhancing their reasoning abilities through continual interaction and feedback. CLUs are built on an architecture that allows a reasoning model to maintain and evolve a dynamic knowledge repository: a General Knowledge Space for broad, reusable insights and a Prompt-Specific Knowledge Space for task-specific learning. Through goal-driven interactions, CLUs iteratively refine these knowledge spaces, enabling the system to adapt dynamically to complex tasks, extract nuanced insights, and build upon past experiences autonomously. We demonstrate CLUs' effectiveness through a cryptographic reasoning task, where they continuously evolve their understanding through feedback to uncover hidden transformation rules. While conventional models struggle to grasp underlying logic, CLUs excel by engaging in an iterative, goal-oriented process. Specialized components-handling knowledge retrieval, prompt generation, and feedback analysis-work together within a reinforcing feedback loop. This approach allows CLUs to retain the memory of past failures and successes, adapt autonomously, and apply sophisticated reasoning effectively, continually learning from mistakes while also building on breakthroughs.

Title: Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning

• Authors: Hyun Ryu, Gyeongman Kim, Hyemin S. Lee, Eunho Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08047
• Pdf URL: https://arxiv.org/pdf/2410.08047
• Copy Paste: [[2410.08047]] Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning(https://arxiv.org/abs/2410.08047)
• Keywords: large language model
• Abstract: Complex logical reasoning tasks require a long sequence of reasoning, which a large language model (LLM) with chain-of-thought prompting still falls short. To alleviate this issue, neurosymbolic approaches incorporate a symbolic solver. Specifically, an LLM only translates a natural language problem into a satisfiability (SAT) problem that consists of first-order logic formulas, and a sound symbolic solver returns a mathematically correct solution. However, we discover that LLMs have difficulties to capture complex logical semantics hidden in the natural language during translation. To resolve this limitation, we propose a Compositional First-Order Logic Translation. An LLM first parses a natural language sentence into newly defined logical dependency structures that consist of an atomic subsentence and its dependents, then sequentially translate the parsed subsentences. Since multiple logical dependency structures and sequential translations are possible for a single sentence, we also introduce two Verification algorithms to ensure more reliable results. We utilize an SAT solver to rigorously compare semantics of generated first-order logic formulas and select the most probable one. We evaluate the proposed method, dubbed CLOVER, on seven logical reasoning benchmarks and show that it outperforms the previous neurosymbolic approaches and achieves new state-of-the-art results.

Title: VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifiers

• Authors: Jianing Qi, Hao Tang, Zhigang Zhu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08048
• Pdf URL: https://arxiv.org/pdf/2410.08048
• Copy Paste: [[2410.08048]] VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifiers(https://arxiv.org/abs/2410.08048)
• Keywords: robust, large language model
• Abstract: Recent advancements in test time compute, particularly through the use of verifier models, have significantly enhanced the reasoning capabilities of Large Language Models (LLMs). This generator-verifier approach closely resembles the actor-critic framework in reinforcement learning (RL). However, current verifier models in LLMs often rely on supervised fine-tuning without temporal difference learning such as Q-learning. This paper introduces VerifierQ, a novel approach that integrates Offline Q-learning into LLM verifier models. We address three key challenges in applying Q-learning to LLMs: (1) handling utterance-level Markov Decision Processes (MDPs), (2) managing large action spaces, and (3) mitigating overestimation bias. VerifierQ introduces a modified Bellman update for bounded Q-values, incorporates Implicit Q-learning (IQL) for efficient action space management, and integrates a novel Conservative Q-learning (CQL) formulation for balanced Q-value estimation. Our method enables parallel Q-value computation and improving training efficiency. While recent work has explored RL techniques like MCTS for generators, VerifierQ is among the first to investigate the verifier (critic) aspect in LLMs through Q-learning. This integration of RL principles into verifier models complements existing advancements in generator techniques, potentially enabling more robust and adaptive reasoning in LLMs. Experimental results on mathematical reasoning tasks demonstrate VerifierQ's superior performance compared to traditional supervised fine-tuning approaches, with improvements in efficiency, accuracy and robustness. By enhancing the synergy between generation and evaluation capabilities, VerifierQ contributes to the ongoing evolution of AI systems in addressing complex cognitive tasks across various domains.

Title: Scaling Up Your Kernels: Large Kernel Design in ConvNets towards Universal Representations

• Authors: Yiyuan Zhang, Xiaohan Ding, Xiangyu Yue
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08049
• Pdf URL: https://arxiv.org/pdf/2410.08049
• Copy Paste: [[2410.08049]] Scaling Up Your Kernels: Large Kernel Design in ConvNets towards Universal Representations(https://arxiv.org/abs/2410.08049)
• Keywords: transformer
• Abstract: This paper proposes the paradigm of large convolutional kernels in designing modern Convolutional Neural Networks (ConvNets). We establish that employing a few large kernels, instead of stacking multiple smaller ones, can be a superior design strategy. Our work introduces a set of architecture design guidelines for large-kernel ConvNets that optimize their efficiency and performance. We propose the UniRepLKNet architecture, which offers systematical architecture design principles specifically crafted for large-kernel ConvNets, emphasizing their unique ability to capture extensive spatial information without deep layer stacking. This results in a model that not only surpasses its predecessors with an ImageNet accuracy of 88.0%, an ADE20K mIoU of 55.6%, and a COCO box AP of 56.4% but also demonstrates impressive scalability and performance on various modalities such as time-series forecasting, audio, point cloud, and video recognition. These results indicate the universal modeling abilities of large-kernel ConvNets with faster inference speed compared with vision transformers. Our findings reveal that large-kernel ConvNets possess larger effective receptive fields and a higher shape bias, moving away from the texture bias typical of smaller-kernel CNNs. All codes and models are publicly available at this https URL promoting further research and development in the community.

Title: A Target-Aware Analysis of Data Augmentation for Hate Speech Detection

• Authors: Camilla Casula, Sara Tonelli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08053
• Pdf URL: https://arxiv.org/pdf/2410.08053
• Copy Paste: [[2410.08053]] A Target-Aware Analysis of Data Augmentation for Hate Speech Detection(https://arxiv.org/abs/2410.08053)
• Keywords: fair, generative
• Abstract: Hate speech is one of the main threats posed by the widespread use of social networks, despite efforts to limit it. Although attention has been devoted to this issue, the lack of datasets and case studies centered around scarcely represented phenomena, such as ableism or ageism, can lead to hate speech detection systems that do not perform well on underrepresented identity groups. Given the unpreceded capabilities of LLMs in producing high-quality data, we investigate the possibility of augmenting existing data with generative language models, reducing target imbalance. We experiment with augmenting 1,000 posts from the Measuring Hate Speech corpus, an English dataset annotated with target identity information, adding around 30,000 synthetic examples using both simple data augmentation methods and different types of generative models, comparing autoregressive and sequence-to-sequence approaches. We find traditional DA methods to often be preferable to generative models, but the combination of the two tends to lead to the best results. Indeed, for some hate categories such as origin, religion, and disability, hate speech classification using augmented data for training improves by more than 10% F1 over the no augmentation baseline. This work contributes to the development of systems for hate speech detection that are not only better performing but also fairer and more inclusive towards targets that have been neglected so far.

Title: Reversible Decoupling Network for Single Image Reflection Removal

• Authors: Hao Zhao, Mingjia Li, Qiming Hu, Xiaojie Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08063
• Pdf URL: https://arxiv.org/pdf/2410.08063
• Copy Paste: [[2410.08063]] Reversible Decoupling Network for Single Image Reflection Removal(https://arxiv.org/abs/2410.08063)
• Keywords: secure
• Abstract: Recent deep-learning-based approaches to single-image reflection removal have shown promising advances, primarily for two reasons: 1) the utilization of recognition-pretrained features as inputs, and 2) the design of dual-stream interaction networks. However, according to the Information Bottleneck principle, high-level semantic clues tend to be compressed or discarded during layer-by-layer propagation. Additionally, interactions in dual-stream networks follow a fixed pattern across different layers, limiting overall performance. To address these limitations, we propose a novel architecture called Reversible Decoupling Network (RDNet), which employs a reversible encoder to secure valuable information while flexibly decoupling transmission- and reflection-relevant features during the forward pass. Furthermore, we customize a transmission-rate-aware prompt generator to dynamically calibrate features, further boosting performance. Extensive experiments demonstrate the superiority of RDNet over existing SOTA methods on five widely-adopted benchmark datasets. Our code will be made publicly available.

Title: Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

• Authors: Shenao Zhang, Zhihan Liu, Boyi Liu, Yufeng Zhang, Yingxiang Yang, Yongfei Liu, Liyu Chen, Tao Sun, Zhaoran Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08067
• Pdf URL: https://arxiv.org/pdf/2410.08067
• Copy Paste: [[2410.08067]] Reward-Augmented Data Enhances Direct Preference Alignment of LLMs(https://arxiv.org/abs/2410.08067)
• Keywords: large language model
• Abstract: Preference alignment in Large Language Models (LLMs) has significantly improved their ability to adhere to human instructions and intentions. However, existing direct alignment algorithms primarily focus on relative preferences and often overlook the qualitative aspects of responses. Striving to maximize the implicit reward gap between the chosen and the slightly inferior rejected responses can cause overfitting and unnecessary unlearning of the high-quality rejected responses. The unawareness of the reward scores also drives the LLM to indiscriminately favor the low-quality chosen responses and fail to generalize to responses with the highest rewards, which are sparse in data. To overcome these shortcomings, our study introduces reward-conditioned LLM policies that discern and learn from the entire spectrum of response quality within the dataset, helping extrapolate to more optimal regions. We propose an effective yet simple data relabeling method that conditions the preference pairs on quality scores to construct a reward-augmented dataset. This dataset is easily integrated with existing direct alignment algorithms and is applicable to any preference dataset. The experimental results across instruction-following benchmarks including AlpacaEval, MT-Bench, and Arena-Hard-Auto demonstrate that our approach consistently boosts the performance of DPO by a considerable margin across diverse models. Additionally, our method improves the average accuracy on various academic benchmarks. When applying our method to on-policy data, the resulting DPO model achieves SOTA results on AlpacaEval. Through ablation studies, we demonstrate that our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere dataset expansion. Our code is available at this https URL.

Title: Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models

• Authors: Wenting Tan, Dongxiao Chen, Jieting Xue, Zihao Wang, Taijie Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08068
• Pdf URL: https://arxiv.org/pdf/2410.08068
• Copy Paste: [[2410.08068]] Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models(https://arxiv.org/abs/2410.08068)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit impressive performance across various domains but still struggle with arithmetic reasoning tasks. Recent work shows the effectiveness of prompt design methods in enhancing reasoning capabilities. However, these approaches overlook crucial requirements for prior knowledge of specific concepts, theorems, and tricks to tackle most arithmetic reasoning problems successfully. To address this issue, we propose a novel and effective Teaching-Inspired Integrated Framework, which emulates the instructional process of a teacher guiding students. This method equips LLMs with essential concepts, relevant theorems, and similar problems with analogous solution approaches, facilitating the enhancement of reasoning abilities. Additionally, we introduce two new Chinese datasets, MathMC and MathToF, both with detailed explanations and answers. Experiments are conducted on nine benchmarks which demonstrates that our approach improves the reasoning accuracy of LLMs. With GPT-4 and our framework, we achieve new state-of-the-art performance on four math benchmarks (AddSub, SVAMP, Math23K and AQuA) with accuracies of 98.2% (+3.3%), 93.9% (+0.2%), 94.3% (+7.2%) and 81.1% (+1.2%). Our data and code are available at this https URL.

Title: Unlearning-based Neural Interpretations

• Authors: Ching Lam Choi, Alexandre Duplessis, Serge Belongie
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08069
• Pdf URL: https://arxiv.org/pdf/2410.08069
• Copy Paste: [[2410.08069]] Unlearning-based Neural Interpretations(https://arxiv.org/abs/2410.08069)
• Keywords: robust
• Abstract: Gradient-based interpretations often require an anchor point of comparison to avoid saturation in computing feature importance. We show that current baselines defined using static functions--constant mapping, averaging or blurring--inject harmful colour, texture or frequency assumptions that deviate from model behaviour. This leads to accumulation of irregular gradients, resulting in attribution maps that are biased, fragile and manipulable. Departing from the static approach, we propose UNI to compute an (un)learnable, debiased and adaptive baseline by perturbing the input towards an unlearning direction of steepest ascent. Our method discovers reliable baselines and succeeds in erasing salient features, which in turn locally smooths the high-curvature decision boundaries. Our analyses point to unlearning as a promising avenue for generating faithful, efficient and robust interpretations.

Title: Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Models

• Authors: Vinith M. Suriyakumar, Rohan Alur, Ayush Sekhari, Manish Raghavan, Ashia C. Wilson
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08074
• Pdf URL: https://arxiv.org/pdf/2410.08074
• Copy Paste: [[2410.08074]] Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Models(https://arxiv.org/abs/2410.08074)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models rely on massive, web-scale datasets. Training them from scratch is computationally expensive, and as a result, developers often prefer to make incremental updates to existing models. These updates often compose fine-tuning steps (to learn new concepts or improve model performance) with "unlearning" steps (to "forget" existing concepts, such as copyrighted works or explicit content). In this work, we demonstrate a critical and previously unknown vulnerability that arises in this paradigm: even under benign, non-adversarial conditions, fine-tuning a text-to-image diffusion model on seemingly unrelated images can cause it to "relearn" concepts that were previously "unlearned." We comprehensively investigate the causes and scope of this phenomenon, which we term concept resurgence, by performing a series of experiments which compose "mass concept erasure" (the current state of the art for unlearning in text-to-image diffusion models (Lu et al., 2024)) with subsequent fine-tuning of Stable Diffusion v1.4. Our findings underscore the fragility of composing incremental model updates, and raise serious new concerns about current approaches to ensuring the safety and alignment of text-to-image diffusion models.

Title: Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study over Open-ended Question Answering

• Authors: Yuan Sui, Bryan Hooi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08085
• Pdf URL: https://arxiv.org/pdf/2410.08085
• Copy Paste: [[2410.08085]] Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study over Open-ended Question Answering(https://arxiv.org/abs/2410.08085)
• Keywords: large language model
• Abstract: Recent works integrating Knowledge Graphs (KGs) have led to promising improvements in enhancing reasoning accuracy of Large Language Models (LLMs). However, current benchmarks mainly focus on closed tasks, leaving a gap in the assessment of more complex, real-world scenarios. This gap has also obscured the evaluation of KGs' potential to mitigate the problem of hallucination in LLMs. To fill the gap, we introduce OKGQA, a new benchmark specifically designed to assess LLMs enhanced with KGs under open-ended, real-world question answering scenarios. OKGQA is designed to closely reflect the complexities of practical applications using questions from different types, and incorporates specific metrics to measure both the reduction in hallucinations and the enhancement in reasoning capabilities. To consider the scenario in which KGs may have varying levels of mistakes, we further propose another experiment setting OKGQA-P to assess model performance when the semantics and structure of KGs are deliberately perturbed and contaminated. OKGQA aims to (1) explore whether KGs can make LLMs more trustworthy in an open-ended setting, and (2) conduct a comparative analysis to shed light on methods and future directions for leveraging KGs to reduce LLMs' hallucination. We believe that this study can facilitate a more complete performance comparison and encourage continuous improvement in integrating KGs with LLMs.

Title: Distribution Guidance Network for Weakly Supervised Point Cloud Semantic Segmentation

• Authors: Zhiyi Pan, Wei Gao, Shan Liu, Ge Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08091
• Pdf URL: https://arxiv.org/pdf/2410.08091
• Copy Paste: [[2410.08091]] Distribution Guidance Network for Weakly Supervised Point Cloud Semantic Segmentation(https://arxiv.org/abs/2410.08091)
• Keywords: segmentation
• Abstract: Despite alleviating the dependence on dense annotations inherent to fully supervised methods, weakly supervised point cloud semantic segmentation suffers from inadequate supervision signals. In response to this challenge, we introduce a novel perspective that imparts auxiliary constraints by regulating the feature space under weak supervision. Our initial investigation identifies which distributions accurately characterize the feature space, subsequently leveraging this priori to guide the alignment of the weakly supervised embeddings. Specifically, we analyze the superiority of the mixture of von Mises-Fisher distributions (moVMF) among several common distribution candidates. Accordingly, we develop a Distribution Guidance Network (DGNet), which comprises a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined latent space. Extensive experiments validate the rationality and effectiveness of our distribution choice and network design. Consequently, DGNet achieves state-of-the-art performance under multiple datasets and various weakly supervised settings.

Title: UW-SDF: Exploiting Hybrid Geometric Priors for Neural SDF Reconstruction from Underwater Multi-view Monocular Images

• Authors: Zeyu Chen, Jingyi Tang, Gu Wang, Shengquan Li, Xinghui Li, Xiangyang Ji, Xiu Li
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.08092
• Pdf URL: https://arxiv.org/pdf/2410.08092
• Copy Paste: [[2410.08092]] UW-SDF: Exploiting Hybrid Geometric Priors for Neural SDF Reconstruction from Underwater Multi-view Monocular Images(https://arxiv.org/abs/2410.08092)
• Keywords: segmentation
• Abstract: Due to the unique characteristics of underwater environments, accurate 3D reconstruction of underwater objects poses a challenging problem in tasks such as underwater exploration and mapping. Traditional methods that rely on multiple sensor data for 3D reconstruction are time-consuming and face challenges in data acquisition in underwater scenarios. We propose UW-SDF, a framework for reconstructing target objects from multi-view underwater images based on neural SDF. We introduce hybrid geometric priors to optimize the reconstruction process, markedly enhancing the quality and efficiency of neural SDF reconstruction. Additionally, to address the challenge of segmentation consistency in multi-view images, we propose a novel few-shot multi-view target segmentation strategy using the general-purpose segmentation model (SAM), enabling rapid automatic segmentation of unseen objects. Through extensive qualitative and quantitative experiments on diverse datasets, we demonstrate that our proposed method outperforms the traditional underwater 3D reconstruction method and other neural rendering approaches in the field of underwater 3D reconstruction.

Title: CrackSegDiff: Diffusion Probability Model-based Multi-modal Crack Segmentation

• Authors: Xiaoyan Jiang, Licheng Jiang, Anjie Wang, Kaiying Zhu, Yongbin Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08100
• Pdf URL: https://arxiv.org/pdf/2410.08100
• Copy Paste: [[2410.08100]] CrackSegDiff: Diffusion Probability Model-based Multi-modal Crack Segmentation(https://arxiv.org/abs/2410.08100)
• Keywords: extraction, diffusion, transformer, segmentation
• Abstract: Integrating grayscale and depth data in road inspection robots could enhance the accuracy, reliability, and comprehensiveness of road condition assessments, leading to improved maintenance strategies and safer infrastructure. However, these data sources are often compromised by significant background noise from the pavement. Recent advancements in Diffusion Probabilistic Models (DPM) have demonstrated remarkable success in image segmentation tasks, showcasing potent denoising capabilities, as evidenced in studies like SegDiff \cite{amit2021segdiff}. Despite these advancements, current DPM-based segmentors do not fully capitalize on the potential of original image data. In this paper, we propose a novel DPM-based approach for crack segmentation, named CrackSegDiff, which uniquely fuses grayscale and range/depth images. This method enhances the reverse diffusion process by intensifying the interaction between local feature extraction via DPM and global feature extraction. Unlike traditional methods that utilize Transformers for global features, our approach employs Vm-unet \cite{ruan2024vm} to efficiently capture long-range information of the original data. The integration of features is further refined through two innovative modules: the Channel Fusion Module (CFM) and the Shallow Feature Compensation Module (SFCM). Our experimental evaluation on the three-class crack image segmentation tasks within the FIND dataset demonstrates that CrackSegDiff outperforms state-of-the-art methods, particularly excelling in the detection of shallow cracks. Code is available at this https URL.

Title: Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining

• Authors: Tianyi Bai, Ling Yang, Zhen Hao Wong, Jiahui Peng, Xinlin Zhuang, Chi Zhang, Lijun Wu, Qiu Jiantao, Wentao Zhang, Binhang Yuan, Conghui He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08102
• Pdf URL: https://arxiv.org/pdf/2410.08102
• Copy Paste: [[2410.08102]] Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining(https://arxiv.org/abs/2410.08102)
• Keywords: large language model
• Abstract: Efficient data selection is crucial to accelerate the pretraining of large language models (LLMs). While various methods have been proposed to enhance data efficiency, limited research has addressed the inherent conflicts between these approaches to achieve optimal data selection for LLM pretraining. To tackle this problem, we propose a novel multi-agent collaborative data selection mechanism. In this framework, each data selection method serves as an independent agent, and an agent console is designed to dynamically integrate the information from all agents throughout the LLM training process. We conduct extensive empirical studies to evaluate our multi-agent framework. The experimental results demonstrate that our approach significantly improves data efficiency, accelerates convergence in LLM training, and achieves an average performance gain of 10.5% across multiple language model benchmarks compared to the state-of-the-art methods.

Title: What Makes Large Language Models Reason in (Multi-Turn) Code Generation?

• Authors: Kunhao Zheng, Juliette Decugis, Jonas Gehring, Taco Cohen, Benjamin Negrevergne, Gabriel Synnaeve
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08105
• Pdf URL: https://arxiv.org/pdf/2410.08105
• Copy Paste: [[2410.08105]] What Makes Large Language Models Reason in (Multi-Turn) Code Generation?(https://arxiv.org/abs/2410.08105)
• Keywords: large language model
• Abstract: Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.

Title: A Closer Look at Machine Unlearning for Large Language Models

• Authors: Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, Min Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08109
• Pdf URL: https://arxiv.org/pdf/2410.08109
• Copy Paste: [[2410.08109]] A Closer Look at Machine Unlearning for Large Language Models(https://arxiv.org/abs/2410.08109)
• Keywords: privacy, large language model
• Abstract: Large language models (LLMs) may memorize sensitive or copyrighted content, raising privacy and legal concerns. Due to the high cost of retraining from scratch, researchers attempt to employ machine unlearning to remove specific content from LLMs while preserving the overall performance. In this paper, we discuss several issues in machine unlearning for LLMs and provide our insights on possible approaches. To address the issue of inadequate evaluation of model outputs after unlearning, we introduce three additional metrics to evaluate token diversity, sentence semantics, and factual correctness. We then categorize unlearning methods into untargeted and targeted, and discuss their issues respectively. Specifically, the behavior that untargeted unlearning attempts to approximate is unpredictable and may involve hallucinations, and existing regularization is insufficient for targeted unlearning. To alleviate these issues, we propose using the objective of maximizing entropy (ME) for untargeted unlearning and incorporate answer preservation (AP) loss as regularization for targeted unlearning. Experimental results across three scenarios, i.e., fictitious unlearning, continual unlearning, and real-world unlearning, demonstrate the effectiveness of our approaches. The code is available at this https URL.

Title: Active Fourier Auditor for Estimating Distributional Properties of ML Models

• Authors: Ayoub Ajarra, Bishwamittra Ghosh, Debabrota Basu
• Subjects: cs.LG, cs.AI, cs.CY, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.08111
• Pdf URL: https://arxiv.org/pdf/2410.08111
• Copy Paste: [[2410.08111]] Active Fourier Auditor for Estimating Distributional Properties of ML Models(https://arxiv.org/abs/2410.08111)
• Keywords: robust, fair
• Abstract: With the pervasive deployment of Machine Learning (ML) models in real-world applications, verifying and auditing properties of ML models have become a central concern. In this work, we focus on three properties: robustness, individual fairness, and group fairness. We discuss two approaches for auditing ML model properties: estimation with and without reconstruction of the target model under audit. Though the first approach is studied in the literature, the second approach remains unexplored. For this purpose, we develop a new framework that quantifies different properties in terms of the Fourier coefficients of the ML model under audit but does not parametrically reconstruct it. We propose the Active Fourier Auditor (AFA), which queries sample points according to the Fourier coefficients of the ML model, and further estimates the properties. We derive high probability error bounds on AFA's estimates, along with the worst-case lower bounds on the sample complexity to audit them. Numerically we demonstrate on multiple datasets and models that AFA is more accurate and sample-efficient to estimate the properties of interest than the baselines.

Title: Robust AI-Generated Text Detection by Restricted Embeddings

• Authors: Kristian Kuznetsov, Eduard Tulchinskii, Laida Kushnareva, German Magai, Serguei Barannikov, Sergey Nikolenko, Irina Piontkovskaya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08113
• Pdf URL: https://arxiv.org/pdf/2410.08113
• Copy Paste: [[2410.08113]] Robust AI-Generated Text Detection by Restricted Embeddings(https://arxiv.org/abs/2410.08113)
• Keywords: robust, transformer
• Abstract: Growing amount and quality of AI-generated texts makes detecting such content more difficult. In most real-world scenarios, the domain (style and topic) of generated data and the generator model are not known in advance. In this work, we focus on the robustness of classifier-based detectors of AI-generated text, namely their ability to transfer to unseen generators or semantic domains. We investigate the geometry of the embedding space of Transformer-based text encoders and show that clearing out harmful linear subspaces helps to train a robust classifier, ignoring domain-specific spurious features. We investigate several subspace decomposition and feature selection strategies and achieve significant improvements over state of the art methods in cross-domain and cross-generator transfer. Our best approaches for head-wise and coordinate-based subspace removal increase the mean out-of-distribution (OOD) classification score by up to 9% and 14% in particular setups for RoBERTa and BERT embeddings respectively. We release our code and data: this https URL

Title: Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System

• Authors: Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08115
• Pdf URL: https://arxiv.org/pdf/2410.08115
• Copy Paste: [[2410.08115]] Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System(https://arxiv.org/abs/2410.08115)
• Keywords: large language model
• Abstract: Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving, yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods. We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness in LLM-based MAS through LLM training. Optima employs an iterative generate, rank, select, and train paradigm with a reward function balancing task performance, token efficiency, and communication readability. We explore various RL algorithms, including Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid approaches, providing insights into their effectiveness-efficiency trade-offs. We integrate Monte Carlo Tree Search-inspired techniques for DPO data generation, treating conversation turns as tree nodes to explore diverse interaction paths. Evaluated on common multi-agent tasks, including information-asymmetric question answering and complex reasoning, Optima shows consistent and substantial improvements over single-agent baselines and vanilla MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than 10\% tokens on tasks requiring heavy information exchange. Moreover, Optima's efficiency gains open new possibilities for leveraging inference-compute more effectively, leading to improved inference-time scaling laws. By addressing fundamental challenges in LLM-based MAS, Optima shows the potential towards scalable, efficient, and effective MAS (this https URL).

Title: On Barycenter Computation: Semi-Unbalanced Optimal Transport-based Method on Gaussians

• Authors: Ngoc-Hai Nguyen, Dung Le, Hoang-Phi Nguyen, Tung Pham, Nhat Ho
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08117
• Pdf URL: https://arxiv.org/pdf/2410.08117
• Copy Paste: [[2410.08117]] On Barycenter Computation: Semi-Unbalanced Optimal Transport-based Method on Gaussians(https://arxiv.org/abs/2410.08117)
• Keywords: robust
• Abstract: We explore a robust version of the barycenter problem among $n$ centered Gaussian probability measures, termed Semi-Unbalanced Optimal Transport (SUOT)-based Barycenter, wherein the barycenter remains fixed while the others are relaxed using Kullback-Leibler divergence. We develop optimization algorithms on Bures-Wasserstein manifold, named the Exact Geodesic Gradient Descent and Hybrid Gradient Descent algorithms. While the Exact Geodesic Gradient Descent method is based on computing the exact closed form of the first-order derivative of the objective function of the barycenter along a geodesic on the Bures manifold, the Hybrid Gradient Descent method utilizes optimizer components when solving the SUOT problem to replace outlier measures before applying the Riemannian Gradient Descent. We establish the theoretical convergence guarantees for both methods and demonstrate that the Exact Geodesic Gradient Descent algorithm attains a dimension-free convergence rate. Finally, we conduct experiments to compare the normal Wasserstein Barycenter with ours and perform an ablation study.

Title: Medical Image Quality Assessment based on Probability of Necessity and Sufficiency

• Authors: Boyu Chen, Ameenat L. Solebo, Weiye Bao, Paul Taylor
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08118
• Pdf URL: https://arxiv.org/pdf/2410.08118
• Copy Paste: [[2410.08118]] Medical Image Quality Assessment based on Probability of Necessity and Sufficiency(https://arxiv.org/abs/2410.08118)
• Keywords: robust
• Abstract: Medical image quality assessment (MIQA) is essential for reliable medical image analysis. While deep learning has shown promise in this field, current models could be misled by spurious correlations learned from data and struggle with out-of-distribution (OOD) scenarios. To that end, we propose an MIQA framework based on a concept from causal inference: Probability of Necessity and Sufficiency (PNS). PNS measures how likely a set of features is to be both necessary (always present for an outcome) and sufficient (capable of guaranteeing an outcome) for a particular result. Our approach leverages this concept by learning hidden features from medical images with high PNS values for quality prediction. This encourages models to capture more essential predictive information, enhancing their robustness to OOD scenarios. We evaluate our framework on an Anterior Segment Optical Coherence Tomography (AS-OCT) dataset for the MIQA task and experimental results demonstrate the effectiveness of our framework.

Title: CCA-Secure Key-Aggregate Proxy Re-Encryption for Secure Cloud Storage

• Authors: Wei-Hao Chen, Chun-I Fan, Yi-Fan Tseng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.08120
• Pdf URL: https://arxiv.org/pdf/2410.08120
• Copy Paste: [[2410.08120]] CCA-Secure Key-Aggregate Proxy Re-Encryption for Secure Cloud Storage(https://arxiv.org/abs/2410.08120)
• Keywords: secure, protect
• Abstract: The development of cloud services in recent years has mushroomed, for example, Google Drive, Amazon AWS, Microsoft Azure. Merchants can easily use cloud services to open their online shops in a few seconds. Users can easily and quickly connect to the cloud in their own portable devices, and access their personal information effortlessly. Because users store large amounts of data on third-party devices, ensuring data confidentiality, availability and integrity become especially important. Therefore, data protection in cloud storage is the key to the survival of the cloud industry. Fortunately, Proxy Re-Encryption schemes enable users to convert their ciphertext into others ciphertext by using a re-encryption key. This method gracefully transforms the users computational cost to the server. In addition, with C-PREs, users can apply their access control right on the encrypted data. Recently, we lowered the key storage cost of C-PREs to constant size and proposed the first Key-Aggregate Proxy Re-Encryption scheme. In this paper, we further prove that our scheme is a CCA-secure Key-Aggregate Proxy Re-Encryption scheme in the adaptive model without using random oracle. Moreover, we also implement and analyze the Key Aggregate PRE application in the real world scenario.

Title: Heterogeneous Graph Auto-Encoder for CreditCard Fraud Detection

• Authors: Moirangthem Tiken Singh, Rabinder Kumar Prasad, Gurumayum Robert Michael, N K Kaphungkui, N.Hemarjit Singh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08121
• Pdf URL: https://arxiv.org/pdf/2410.08121
• Copy Paste: [[2410.08121]] Heterogeneous Graph Auto-Encoder for CreditCard Fraud Detection(https://arxiv.org/abs/2410.08121)
• Keywords: security
• Abstract: The digital revolution has significantly impacted financial transactions, leading to a notable increase in credit card usage. However, this convenience comes with a trade-off: a substantial rise in fraudulent activities. Traditional machine learning methods for fraud detection often struggle to capture the inherent interconnectedness within financial data. This paper proposes a novel approach for credit card fraud detection that leverages Graph Neural Networks (GNNs) with attention mechanisms applied to heterogeneous graph representations of financial data. Unlike homogeneous graphs, heterogeneous graphs capture intricate relationships between various entities in the financial ecosystem, such as cardholders, merchants, and transactions, providing a richer and more comprehensive data representation for fraud analysis. To address the inherent class imbalance in fraud data, where genuine transactions significantly outnumber fraudulent ones, the proposed approach integrates an autoencoder. This autoencoder, trained on genuine transactions, learns a latent representation and flags deviations during reconstruction as potential fraud. This research investigates two key questions: (1) How effectively can a GNN with an attention mechanism detect and prevent credit card fraud when applied to a heterogeneous graph? (2) How does the efficacy of the autoencoder with attention approach compare to traditional methods? The results are promising, demonstrating that the proposed model outperforms benchmark algorithms such as Graph Sage and FI-GRL, achieving a superior AUC-PR of 0.89 and an F1-score of 0.81. This research significantly advances fraud detection systems and the overall security of financial transactions by leveraging GNNs with attention mechanisms and addressing class imbalance through an autoencoder.

Title: PP-GWAS: Privacy Preserving Multi-Site Genome-wide Association Studies

• Authors: Arjhun Swaminathan, Anika Hannemann, Ali Burak Ünal, Nico Pfeifer, Mete Akgün
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.08122
• Pdf URL: https://arxiv.org/pdf/2410.08122
• Copy Paste: [[2410.08122]] PP-GWAS: Privacy Preserving Multi-Site Genome-wide Association Studies(https://arxiv.org/abs/2410.08122)
• Keywords: secure, security, privacy, robust
• Abstract: Genome-wide association studies are pivotal in understanding the genetic underpinnings of complex traits and diseases. Collaborative, multi-site GWAS aim to enhance statistical power but face obstacles due to the sensitive nature of genomic data sharing. Current state-of-the-art methods provide a privacy-focused approach utilizing computationally expensive methods such as Secure Multi-Party Computation and Homomorphic Encryption. In this context, we present a novel algorithm PP-GWAS designed to improve upon existing standards in terms of computational efficiency and scalability without sacrificing data privacy. This algorithm employs randomized encoding within a distributed architecture to perform stacked ridge regression on a Linear Mixed Model to ensure rigorous analysis. Experimental evaluation with real world and synthetic data indicates that PP-GWAS can achieve computational speeds twice as fast as similar state-of-the-art algorithms while using lesser computational resources, all while adhering to a robust security model that caters to an all-but-one semi-honest adversary setting. We have assessed its performance using various datasets, emphasizing its potential in facilitating more efficient and private genomic analyses.

Title: Mars: Situated Inductive Reasoning in an Open-World Environment

• Authors: Xiaojuan Tang, Jiaqi Li, Yitao Liang, Song-chun Zhu, Muhan Zhang, Zilong Zheng
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08126
• Pdf URL: https://arxiv.org/pdf/2410.08126
• Copy Paste: [[2410.08126]] Mars: Situated Inductive Reasoning in an Open-World Environment(https://arxiv.org/abs/2410.08126)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) trained on massive corpora have shown remarkable success in knowledge-intensive tasks. Yet, most of them rely on pre-stored knowledge. Inducing new general knowledge from a specific environment and performing reasoning with the acquired knowledge -- \textit{situated inductive reasoning}, is crucial and challenging for machine intelligence. In this paper, we design Mars, an interactive environment devised for situated inductive reasoning. It introduces counter-commonsense game mechanisms by modifying terrain, survival setting and task dependency while adhering to certain principles. In Mars, agents need to actively interact with their surroundings, derive useful rules and perform decision-making tasks in specific contexts. We conduct experiments on various RL-based and LLM-based methods, finding that they all struggle on this challenging situated inductive reasoning benchmark. Furthermore, we explore \textit{Induction from Reflection}, where we instruct agents to perform inductive reasoning from history trajectory. The superior performance underscores the importance of inductive reasoning in Mars. Through Mars, we aim to galvanize advancements in situated inductive reasoning and set the stage for developing the next generation of AI systems that can reason in an adaptive and context-sensitive way.

Title: Think Beyond Size: Dynamic Prompting for More Effective Reasoning

• Authors: Kamesh R
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08130
• Pdf URL: https://arxiv.org/pdf/2410.08130
• Copy Paste: [[2410.08130]] Think Beyond Size: Dynamic Prompting for More Effective Reasoning(https://arxiv.org/abs/2410.08130)
• Keywords: large language model
• Abstract: This paper presents Dynamic Prompting, a novel framework aimed at improving the reasoning capabilities of Large Language Models (LLMs). In contrast to conventional static prompting methods, Dynamic Prompting enables the adaptive modification of prompt sequences and step counts based on real-time task complexity and model performance. This dynamic adaptation facilitates more efficient problem-solving, particularly in smaller models, by reducing hallucinations and repetitive cycles. Our empirical evaluations demonstrate that Dynamic Prompting allows smaller LLMs to perform competitively with much larger models, thereby challenging the conventional emphasis on model size as the primary determinant of reasoning efficacy.

Title: Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction

• Authors: Jarrid Rector-Brooks, Mohsin Hasan, Zhangzhi Peng, Zachary Quinn, Chenghao Liu, Sarthak Mittal, Nouha Dziri, Michael Bronstein, Yoshua Bengio, Pranam Chatterjee, Alexander Tong, Avishek Joey Bose
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08134
• Pdf URL: https://arxiv.org/pdf/2410.08134
• Copy Paste: [[2410.08134]] Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction(https://arxiv.org/abs/2410.08134)
• Keywords: diffusion, generative
• Abstract: Generative modeling of discrete data underlies important applications spanning text-based agents like ChatGPT to the design of the very building blocks of life in protein sequences. However, application domains need to exert control over the generated data by steering the generative process - typically via RLHF - to satisfy a specified property, reward, or affinity metric. In this paper, we study the problem of steering Masked Diffusion Models (MDMs), a recent class of discrete diffusion models that offer a compelling alternative to traditional autoregressive models. We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference by learning to sample from a target Bayesian posterior. Our DDPP framework leads to a family of three novel objectives that are all simulation-free, and thus scalable while applying to general non-differentiable reward functions. Empirically, we instantiate DDPP by steering MDMs to perform class-conditional pixel-level image modeling, RLHF-based alignment of MDMs using text-based rewards, and finetuning protein language models to generate more diverse secondary structures and shorter proteins. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.

Title: DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory

• Authors: Yutong Wang, Jiali Zeng, Xuebo Liu, Derek F. Wong, Fandong Meng, Jie Zhou, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08143
• Pdf URL: https://arxiv.org/pdf/2410.08143
• Copy Paste: [[2410.08143]] DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory(https://arxiv.org/abs/2410.08143)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved reasonable quality improvements in machine translation (MT). However, most current research on MT-LLMs still faces significant challenges in maintaining translation consistency and accuracy when processing entire documents. In this paper, we introduce DelTA, a Document-levEL Translation Agent designed to overcome these limitations. DelTA features a multi-level memory structure that stores information across various granularities and spans, including Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-Term Memory, which are continuously retrieved and updated by auxiliary LLM-based components. Experimental results indicate that DelTA significantly outperforms strong baselines in terms of translation consistency and quality across four open/closed-source LLMs and two representative document translation datasets, achieving an increase in consistency scores by up to 4.58 percentage points and in COMET scores by up to 3.16 points on average. DelTA employs a sentence-by-sentence translation strategy, ensuring no sentence omissions and offering a memory-efficient solution compared to the mainstream method. Furthermore, DelTA improves pronoun translation accuracy, and the summary component of the agent also shows promise as a tool for query-based summarization tasks. We release our code and data at this https URL.

Title: Insight Over Sight? Exploring the Vision-Knowledge Conflicts in Multimodal LLMs

• Authors: Xiaoyuan Liu, Wenxuan Wang, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Pinjia He, Zhaopeng Tu
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08145
• Pdf URL: https://arxiv.org/pdf/2410.08145
• Copy Paste: [[2410.08145]] Insight Over Sight? Exploring the Vision-Knowledge Conflicts in Multimodal LLMs(https://arxiv.org/abs/2410.08145)
• Keywords: large language model
• Abstract: This paper explores the problem of commonsense-level vision-knowledge conflict in Multimodal Large Language Models (MLLMs), where visual information contradicts model's internal commonsense knowledge (see Figure 1). To study this issue, we introduce an automated pipeline, augmented with human-in-the-loop quality control, to establish a benchmark aimed at simulating and assessing the conflicts in MLLMs. Utilizing this pipeline, we have crafted a diagnostic benchmark comprising 374 original images and 1,122 high-quality question-answer (QA) pairs. This benchmark covers two types of conflict target and three question difficulty levels, providing a thorough assessment tool. Through this benchmark, we evaluate the conflict-resolution capabilities of nine representative MLLMs across various model families and find a noticeable over-reliance on textual queries. Drawing on these findings, we propose a novel prompting strategy, "Focus-on-Vision" (FoV), which markedly enhances MLLMs' ability to favor visual data over conflicting textual knowledge. Our detailed analysis and the newly proposed strategy significantly advance the understanding and mitigating of vision-knowledge conflicts in MLLMs. The data and code are made publicly available.

Title: Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning

• Authors: Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, Aviral Kumar
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08146
• Pdf URL: https://arxiv.org/pdf/2410.08146
• Copy Paste: [[2410.08146]] Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning(https://arxiv.org/abs/2410.08146)
• Keywords: large language model
• Abstract: A promising approach for improving reasoning in large language models is to use process reward models (PRMs). PRMs provide feedback at each step of a multi-step reasoning trace, potentially improving credit assignment over outcome reward models (ORMs) that only provide feedback at the final step. However, collecting dense, per-step human labels is not scalable, and training PRMs from automatically-labeled data has thus far led to limited gains. To improve a base policy by running search against a PRM or using it as dense rewards for reinforcement learning (RL), we ask: "How should we design process rewards?". Our key insight is that, to be effective, the process reward for a step should measure progress: a change in the likelihood of producing a correct response in the future, before and after taking the step, corresponding to the notion of step-level advantages in RL. Crucially, this progress should be measured under a prover policy distinct from the base policy. We theoretically characterize the set of good provers and our results show that optimizing process rewards from such provers improves exploration during test-time search and online RL. In fact, our characterization shows that weak prover policies can substantially improve a stronger base policy, which we also observe empirically. We validate our claims by training process advantage verifiers (PAVs) to predict progress under such provers, and show that compared to ORMs, test-time search against PAVs is $>8\%$ more accurate, and $1.5-5\times$ more compute-efficient. Online RL with dense rewards from PAVs enables one of the first results with $5-6\times$ gain in sample efficiency, and $>6\%$ gain in accuracy, over ORMs.

Title: Progressive Autoregressive Video Diffusion Models

• Authors: Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, Yang Zhou
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08151
• Pdf URL: https://arxiv.org/pdf/2410.08151
• Copy Paste: [[2410.08151]] Progressive Autoregressive Video Diffusion Models(https://arxiv.org/abs/2410.08151)
• Keywords: diffusion
• Abstract: Current frontier video diffusion models have demonstrated remarkable results at generating high-quality videos. However, they can only generate short video clips, normally around 10 seconds or 240 frames, due to computation limitations during training. In this work, we show that existing models can be naturally extended to autoregressive video diffusion models without changing the architectures. Our key idea is to assign the latent frames with progressively increasing noise levels rather than a single noise level, which allows for fine-grained condition among the latents and large overlaps between the attention windows. Such progressive video denoising allows our models to autoregressively generate video frames without quality degradation or abrupt scene changes. We present state-of-the-art results on long video generation at 1 minute (1440 frames at 24 FPS). Videos from this paper are available at this https URL.

Title: DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation

• Authors: Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, Shuangfei Zhai
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08159
• Pdf URL: https://arxiv.org/pdf/2410.08159
• Copy Paste: [[2410.08159]] DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation(https://arxiv.org/abs/2410.08159)
• Keywords: diffusion, transformer
• Abstract: Diffusion models have become the dominant approach for visual generation. They are trained by denoising a Markovian process that gradually adds noise to the input. We argue that the Markovian property limits the models ability to fully utilize the generation trajectory, leading to inefficiencies during training and inference. In this paper, we propose DART, a transformer-based model that unifies autoregressive (AR) and diffusion within a non-Markovian framework. DART iteratively denoises image patches spatially and spectrally using an AR model with the same architecture as standard language models. DART does not rely on image quantization, enabling more effective image modeling while maintaining flexibility. Furthermore, DART seamlessly trains with both text and image data in a unified model. Our approach demonstrates competitive performance on class-conditioned and text-to-image generation tasks, offering a scalable, efficient alternative to traditional diffusion models. Through this unified framework, DART sets a new benchmark for scalable, high-quality image synthesis.

Title: ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion

• Authors: Zitian Zhang, Frédéric Fortier-Chouinard, Mathieu Garon, Anand Bhattad, Jean-François Lalonde
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08168
• Pdf URL: https://arxiv.org/pdf/2410.08168
• Copy Paste: [[2410.08168]] ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion(https://arxiv.org/abs/2410.08168)
• Keywords: diffusion, generative
• Abstract: We present ZeroComp, an effective zero-shot 3D object compositing approach that does not require paired composite-scene images during training. Our method leverages ControlNet to condition from intrinsic images and combines it with a Stable Diffusion model to utilize its scene priors, together operating as an effective rendering engine. During training, ZeroComp uses intrinsic images based on geometry, albedo, and masked shading, all without the need for paired images of scenes with and without composite objects. Once trained, it seamlessly integrates virtual 3D objects into scenes, adjusting shading to create realistic composites. We developed a high-quality evaluation dataset and demonstrate that ZeroComp outperforms methods using explicit lighting estimations and generative techniques in quantitative and human perception benchmarks. Additionally, ZeroComp extends to real and outdoor image compositing, even when trained solely on synthetic indoor data, showcasing its effectiveness in image compositing.

Title: Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models

• Authors: Qingni Wang, Tiantian Geng, Zhiyuan Wang, Teng Wang, Bo Fu, Feng Zheng
• Subjects: cs.CL, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.08174
• Pdf URL: https://arxiv.org/pdf/2410.08174
• Copy Paste: [[2410.08174]] Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models(https://arxiv.org/abs/2410.08174)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) exhibit promising advancements across various tasks, yet they still encounter significant trustworthiness issues. Prior studies apply Split Conformal Prediction (SCP) in language modeling to construct prediction sets with statistical guarantees. However, these methods typically rely on internal model logits or are restricted to multiple-choice settings, which hampers their generalizability and adaptability in dynamic, open-ended environments. In this paper, we introduce TRON, a two-step framework for risk control and assessment, applicable to any MLLM that supports sampling in both open-ended and closed-ended scenarios. TRON comprises two main components: (1) a novel conformal score to sample response sets of minimum size, and (2) a nonconformity score to identify high-quality responses based on self-consistency theory, controlling the error rates by two specific risk levels. Furthermore, we investigate semantic redundancy in prediction sets within open-ended contexts for the first time, leading to a promising evaluation metric for MLLMs based on average set size. Our comprehensive experiments across four Video Question-Answering (VideoQA) datasets utilizing eight MLLMs show that TRON achieves desired error rates bounded by two user-specified risk levels. Additionally, deduplicated prediction sets maintain adaptiveness while being more efficient and stable for risk assessment under different risk levels.

Title: RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image

• Authors: Xiaoxue Chen, Jv Zheng, Hao Huang, Haoran Xu, Weihao Gu, Kangliang Chen, He xiang, Huan-ang Gao, Hao Zhao, Guyue Zhou, Yaqin Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08181
• Pdf URL: https://arxiv.org/pdf/2410.08181
• Copy Paste: [[2410.08181]] RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image(https://arxiv.org/abs/2410.08181)
• Keywords: generative
• Abstract: The generation of high-quality 3D car assets is essential for various applications, including video games, autonomous driving, and virtual reality. Current 3D generation methods utilizing NeRF or 3D-GS as representations for 3D objects, generate a Lambertian object under fixed lighting and lack separated modelings for material and global illumination. As a result, the generated assets are unsuitable for relighting under varying lighting conditions, limiting their applicability in downstream tasks. To address this challenge, we propose a novel relightable 3D object generative framework that automates the creation of 3D car assets, enabling the swift and accurate reconstruction of a vehicle's geometry, texture, and material properties from a single input image. Our approach begins with introducing a large-scale synthetic car dataset comprising over 1,000 high-precision 3D vehicle models. We represent 3D objects using global illumination and relightable 3D Gaussian primitives integrating with BRDF parameters. Building on this representation, we introduce a feed-forward model that takes images as input and outputs both relightable 3D Gaussians and global illumination parameters. Experimental results demonstrate that our method produces photorealistic 3D car assets that can be seamlessly integrated into road scenes with different illuminations, which offers substantial practical benefits for industrial applications.

Title: Scaling Laws For Diffusion Transformers

• Authors: Zhengyang Liang, Hao He, Ceyuan Yang, Bo Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08184
• Pdf URL: https://arxiv.org/pdf/2410.08184
• Copy Paste: [[2410.08184]] Scaling Laws For Diffusion Transformers(https://arxiv.org/abs/2410.08184)
• Keywords: diffusion, transformer
• Abstract: Diffusion transformers (DiT) have already achieved appealing synthesis and scaling properties in content recreation, e.g., image and video generation. However, scaling laws of DiT are less explored, which usually offer precise predictions regarding optimal model size and data requirements given a specific compute budget. Therefore, experiments across a broad range of compute budgets, from 1e17 to 6e18 FLOPs are conducted to confirm the existence of scaling laws in DiT for the first time. Concretely, the loss of pretraining DiT also follows a power-law relationship with the involved compute. Based on the scaling law, we can not only determine the optimal model size and required data but also accurately predict the text-to-image generation loss given a model with 1B parameters and a compute budget of 1e21 FLOPs. Additionally, we also demonstrate that the trend of pre-training loss matches the generation performances (e.g., FID), even across various datasets, which complements the mapping from compute to synthesis quality and thus provides a predictable benchmark that assesses model performance and data quality at a reduced cost.

Title: DifFRelight: Diffusion-Based Facial Performance Relighting

• Authors: Mingming He, Pascal Clausen, Ahmet Levent Taşel, Li Ma, Oliver Pilarski, Wenqi Xian, Laszlo Rikker, Xueming Yu, Ryan Burgert, Ning Yu, Paul Debevec
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.08188
• Pdf URL: https://arxiv.org/pdf/2410.08188
• Copy Paste: [[2410.08188]] DifFRelight: Diffusion-Based Facial Performance Relighting(https://arxiv.org/abs/2410.08188)
• Keywords: diffusion
• Abstract: We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. Leveraging a subject-specific dataset containing diverse facial expressions captured under various lighting conditions, including flat-lit and one-light-at-a-time (OLAT) scenarios, we train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. Our framework includes spatially-aligned conditioning of flat-lit captures and random noise, along with integrated lighting information for global control, utilizing prior knowledge from the pre-trained Stable Diffusion model. This model is then applied to dynamic facial performances captured in a consistent flat-lit environment and reconstructed for novel-view synthesis using a scalable dynamic 3D Gaussian Splatting method to maintain quality and consistency in the relit results. In addition, we introduce unified lighting control by integrating a novel area lighting representation with directional lighting, allowing for joint adjustments in light size and direction. We also enable high dynamic range imaging (HDRI) composition using multiple directional lights to produce dynamic sequences under complex lighting conditions. Our evaluations demonstrate the models efficiency in achieving precise lighting control and generalizing across various facial expressions while preserving detailed features such as skintexture andhair. The model accurately reproduces complex lighting effects like eye reflections, subsurface scattering, self-shadowing, and translucency, advancing photorealism within our framework.

Title: Poison-splat: Computation Cost Attack on 3D Gaussian Splatting

• Authors: Jiahao Lu, Yifan Zhang, Qiuhong Shen, Xinchao Wang, Shuicheng Yan
• Subjects: cs.CV, cs.CR, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08190
• Pdf URL: https://arxiv.org/pdf/2410.08190
• Copy Paste: [[2410.08190]] Poison-splat: Computation Cost Attack on 3D Gaussian Splatting(https://arxiv.org/abs/2410.08190)
• Keywords: security, attack
• Abstract: 3D Gaussian splatting (3DGS), known for its groundbreaking performance and efficiency, has become a dominant 3D representation and brought progress to many 3D vision tasks. However, in this work, we reveal a significant security vulnerability that has been largely overlooked in 3DGS: the computation cost of training 3DGS could be maliciously tampered by poisoning the input data. By developing an attack named Poison-splat, we reveal a novel attack surface where the adversary can poison the input images to drastically increase the computation memory and time needed for 3DGS training, pushing the algorithm towards its worst computation complexity. In extreme cases, the attack can even consume all allocable memory, leading to a Denial-of-Service (DoS) that disrupts servers, resulting in practical damages to real-world 3DGS service vendors. Such a computation cost attack is achieved by addressing a bi-level optimization problem through three tailored strategies: attack objective approximation, proxy model rendering, and optional constrained optimization. These strategies not only ensure the effectiveness of our attack but also make it difficult to defend with simple defensive measures. We hope the revelation of this novel attack surface can spark attention to this crucial yet overlooked vulnerability of 3DGS systems.

Title: HybridBooth: Hybrid Prompt Inversion for Efficient Subject-Driven Generation

• Authors: Shanyan Guan, Yanhao Ge, Ying Tai, Jian Yang, Wei Li, Mingyu You
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08192
• Pdf URL: https://arxiv.org/pdf/2410.08192
• Copy Paste: [[2410.08192]] HybridBooth: Hybrid Prompt Inversion for Efficient Subject-Driven Generation(https://arxiv.org/abs/2410.08192)
• Keywords: robust, diffusion
• Abstract: Recent advancements in text-to-image diffusion models have shown remarkable creative capabilities with textual prompts, but generating personalized instances based on specific subjects, known as subject-driven generation, remains challenging. To tackle this issue, we present a new hybrid framework called HybridBooth, which merges the benefits of optimization-based and direct-regression methods. HybridBooth operates in two stages: the Word Embedding Probe, which generates a robust initial word embedding using a fine-tuned encoder, and the Word Embedding Refinement, which further adapts the encoder to specific subject images by optimizing key parameters. This approach allows for effective and fast inversion of visual concepts into textual embedding, even from a single image, while maintaining the model's generalization capabilities.

Title: GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment

• Authors: Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, Sumitra Ganesh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08193
• Pdf URL: https://arxiv.org/pdf/2410.08193
• Copy Paste: [[2410.08193]] GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment(https://arxiv.org/abs/2410.08193)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit impressive capabilities but require careful alignment with human preferences. Traditional training-time methods finetune LLMs using human preference datasets but incur significant training costs and require repeated training to handle diverse user preferences. Test-time alignment methods address this by using reward models (RMs) to guide frozen LLMs without retraining. However, existing test-time approaches rely on trajectory-level RMs which are designed to evaluate complete responses, making them unsuitable for autoregressive text generation that requires computing next-token rewards from partial responses. To address this, we introduce GenARM, a test-time alignment approach that leverages the Autoregressive Reward Model--a novel reward parametrization designed to predict next-token rewards for efficient and effective autoregressive generation. Theoretically, we demonstrate that this parametrization can provably guide frozen LLMs toward any distribution achievable by traditional RMs within the KL-regularized reinforcement learning framework. Experimental results show that GenARM significantly outperforms prior test-time alignment baselines and matches the performance of training-time methods. Additionally, GenARM enables efficient weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high costs of training larger models. Furthermore, GenARM supports multi-objective alignment, allowing real-time trade-offs between preference dimensions and catering to diverse user preferences without retraining.

Title: MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

• Authors: Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08196
• Pdf URL: https://arxiv.org/pdf/2410.08196
• Copy Paste: [[2410.08196]] MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code(https://arxiv.org/abs/2410.08196)
• Keywords: large language model
• Abstract: Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline. The code is released at this https URL .

Title: From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions

• Authors: Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, Ji-Rong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.08197
• Pdf URL: https://arxiv.org/pdf/2410.08197
• Copy Paste: [[2410.08197]] From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions(https://arxiv.org/abs/2410.08197)
• Keywords: robust, large language model
• Abstract: Tool learning enables Large Language Models (LLMs) to interact with external environments by invoking tools, serving as an effective strategy to mitigate the limitations inherent in their pre-training data. In this process, tool documentation plays a crucial role by providing usage instructions for LLMs, thereby facilitating effective tool utilization. This paper concentrates on the critical challenge of bridging the comprehension gap between LLMs and external tools due to the inadequacies and inaccuracies inherent in existing human-centric tool documentation. We propose a novel framework, DRAFT, aimed at Dynamically Refining tool documentation through the Analysis of Feedback and Trails emanating from LLMs' interactions with external tools. This methodology pivots on an innovative trial-and-error approach, consisting of three distinct learning phases: experience gathering, learning from experience, and documentation rewriting, to iteratively enhance the tool documentation. This process is further optimized by implementing a diversity-promoting exploration strategy to ensure explorative diversity and a tool-adaptive termination mechanism to prevent overfitting while enhancing efficiency. Extensive experiments on multiple datasets demonstrate that DRAFT's iterative, feedback-based refinement significantly ameliorates documentation quality, fostering a deeper comprehension and more effective utilization of tools by LLMs. Notably, our analysis reveals that the tool documentation refined via our approach demonstrates robust cross-model generalization capabilities.

Title: Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

• Authors: Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jifeng Dai, Yu Qiao, Xizhou Zhu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.08202
• Pdf URL: https://arxiv.org/pdf/2410.08202
• Copy Paste: [[2410.08202]] Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training(https://arxiv.org/abs/2410.08202)
• Keywords: large language model
• Abstract: The rapid advancement of Large Language Models (LLMs) has led to an influx of efforts to extend their capabilities to multimodal tasks. Among them, growing attention has been focused on monolithic Multimodal Large Language Models (MLLMs) that integrate visual encoding and language decoding into a single LLM. Despite the structural simplicity and deployment-friendliness, training a monolithic MLLM with promising performance still remains challenging. In particular, the popular approaches adopt continuous pre-training to extend a pre-trained LLM to a monolithic MLLM, which suffers from catastrophic forgetting and leads to performance degeneration. In this paper, we aim to overcome this limitation from the perspective of delta tuning. Specifically, our core idea is to embed visual parameters into a pre-trained LLM, thereby incrementally learning visual knowledge from massive data via delta tuning, i.e., freezing the LLM when optimizing the visual parameters. Based on this principle, we present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure. Moreover, we propose an innovative pre-training strategy to maximize the visual capability of Mono-InternVL, namely Endogenous Visual Pre-training (EViP). In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data. To validate our approach, we conduct extensive experiments on 16 benchmarks. Experimental results not only validate the superior performance of Mono-InternVL compared to the state-of-the-art MLLM on 6 multimodal benchmarks, e.g., +113 points over InternVL-1.5 on OCRBench, but also confirm its better deployment efficiency, with first token latency reduced by up to 67%.

Title: Interactive4D: Interactive 4D LiDAR Segmentation

• Authors: Ilya Fradlin, Idil Esen Zulfikar, Kadir Yilmaz, Theodora Kontogianni, Bastian Leibe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.08206
• Pdf URL: https://arxiv.org/pdf/2410.08206
• Copy Paste: [[2410.08206]] Interactive4D: Interactive 4D LiDAR Segmentation(https://arxiv.org/abs/2410.08206)
• Keywords: segmentation
• Abstract: Interactive segmentation has an important role in facilitating the annotation process of future LiDAR datasets. Existing approaches sequentially segment individual objects at each LiDAR scan, repeating the process throughout the entire sequence, which is redundant and ineffective. In this work, we propose interactive 4D segmentation, a new paradigm that allows segmenting multiple objects on multiple LiDAR scans simultaneously, and Interactive4D, the first interactive 4D segmentation model that segments multiple objects on superimposed consecutive LiDAR scans in a single iteration by utilizing the sequential nature of LiDAR data. While performing interactive segmentation, our model leverages the entire space-time volume, leading to more efficient segmentation. Operating on the 4D volume, it directly provides consistent instance IDs over time and also simplifies tracking annotations. Moreover, we show that click simulations are crucial for successful model training on LiDAR point clouds. To this end, we design a click simulation strategy that is better suited for the characteristics of LiDAR data. To demonstrate its accuracy and effectiveness, we evaluate Interactive4D on multiple LiDAR datasets, where Interactive4D achieves a new state-of-the-art by a large margin. Upon acceptance, we will publicly release the code and models at this https URL.

Title: DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models

• Authors: Xiaoxiao He, Ligong Han, Quan Dao, Song Wen, Minhao Bai, Di Liu, Han Zhang, Martin Renqiang Min, Felix Juefei-Xu, Chaowei Tan, Bo Liu, Kang Li, Hongdong Li, Junzhou Huang, Faez Ahmed, Akash Srivastava, Dimitris Metaxas
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08207
• Pdf URL: https://arxiv.org/pdf/2410.08207
• Copy Paste: [[2410.08207]] DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models(https://arxiv.org/abs/2410.08207)
• Keywords: diffusion, generative
• Abstract: Discrete diffusion models have achieved success in tasks like image generation and masked language modeling but face limitations in controlled content editing. We introduce DICE (Discrete Inversion for Controllable Editing), the first approach to enable precise inversion for discrete diffusion models, including multinomial diffusion and masked generative models. By recording noise sequences and masking patterns during the reverse diffusion process, DICE enables accurate reconstruction and flexible editing of discrete data without the need for predefined masks or attention manipulation. We demonstrate the effectiveness of DICE across both image and text domains, evaluating it on models such as VQ-Diffusion, Paella, and RoBERTa. Our results show that DICE preserves high data fidelity while enhancing editing capabilities, offering new opportunities for fine-grained content manipulation in discrete spaces. For project webpage, see this https URL.

Title: SPA: 3D Spatial-Awareness Enables Effective Embodied Representation

• Authors: Haoyi Zhu, Honghui Yang, Yating Wang, Jiange Yang, Limin Wang, Tong He
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.08208
• Pdf URL: https://arxiv.org/pdf/2410.08208
• Copy Paste: [[2410.08208]] SPA: 3D Spatial-Awareness Enables Effective Embodied Representation(https://arxiv.org/abs/2410.08208)
• Keywords: transformer
• Abstract: In this paper, we introduce SPA, a novel representation learning framework that emphasizes the importance of 3D spatial awareness in embodied AI. Our approach leverages differentiable neural rendering on multi-view images to endow a vanilla Vision Transformer (ViT) with intrinsic spatial understanding. We present the most comprehensive evaluation of embodied representation learning to date, covering 268 tasks across 8 simulators with diverse policies in both single-task and language-conditioned multi-task scenarios. The results are compelling: SPA consistently outperforms more than 10 state-of-the-art representation methods, including those specifically designed for embodied AI, vision-centric tasks, and multi-modal applications, while using less training data. Furthermore, we conduct a series of real-world experiments to confirm its effectiveness in practical scenarios. These results highlight the critical role of 3D spatial awareness for embodied representation learning. Our strongest model takes more than 6000 GPU hours to train and we are committed to open-sourcing all code and model weights to foster future research in embodied representation learning. Project Page: this https URL.

Title: Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision

• Authors: Shengcao Cao, Liang-Yan Gui, Yu-Xiong Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.08209
• Pdf URL: https://arxiv.org/pdf/2410.08209
• Copy Paste: [[2410.08209]] Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision(https://arxiv.org/abs/2410.08209)
• Keywords: diffusion, segmentation
• Abstract: Current large multimodal models (LMMs) face challenges in grounding, which requires the model to relate language components to visual entities. Contrary to the common practice that fine-tunes LMMs with additional grounding supervision, we find that the grounding ability can in fact emerge in LMMs trained without explicit grounding supervision. To reveal this emerging grounding, we introduce an "attend-and-segment" method which leverages attention maps from standard LMMs to perform pixel-level segmentation. Furthermore, to enhance the grounding ability, we propose DIFFLMM, an LMM utilizing a diffusion-based visual encoder, as opposed to the standard CLIP visual encoder, and trained with the same weak supervision. Without being constrained by the biases and limited scale of grounding-specific supervision data, our approach is more generalizable and scalable. We achieve competitive performance on both grounding-specific and general visual question answering benchmarks, compared with grounding LMMs and generalist LMMs, respectively. Notably, we achieve a 44.2 grounding mask recall on grounded conversation generation without any grounding supervision, outperforming the extensively supervised model GLaMM. Project page: this https URL.