2025-02-03

Title: STAMP: Scalable Task And Model-agnostic Collaborative Perception

• Authors: Xiangbo Gao, Runsheng Xu, Jiachen Li, Ziran Wang, Zhiwen Fan, Zhengzhong Tu
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.18616
• Pdf URL: https://arxiv.org/pdf/2501.18616
• Copy Paste: [[2501.18616]] STAMP: Scalable Task And Model-agnostic Collaborative Perception(https://arxiv.org/abs/2501.18616)
• Keywords: secure, security
• Abstract: Perception is crucial for autonomous driving, but single-agent perception is often constrained by sensors' physical limitations, leading to degraded performance under severe occlusion, adverse weather conditions, and when detecting distant objects. Multi-agent collaborative perception offers a solution, yet challenges arise when integrating heterogeneous agents with varying model architectures. To address these challenges, we propose STAMP, a scalable task- and model-agnostic, collaborative perception pipeline for heterogeneous agents. STAMP utilizes lightweight adapter-reverter pairs to transform Bird's Eye View (BEV) features between agent-specific and shared protocol domains, enabling efficient feature sharing and fusion. This approach minimizes computational overhead, enhances scalability, and preserves model security. Experiments on simulated and real-world datasets demonstrate STAMP's comparable or superior accuracy to state-of-the-art models with significantly reduced computational costs. As a first-of-its-kind task- and model-agnostic framework, STAMP aims to advance research in scalable and secure mobility systems towards Level 5 autonomy. Our project page is at this https URL and the code is available at this https URL.

Title: DarkMind: Latent Chain-of-Thought Backdoor in Customized LLMs

• Authors: Zhen Guo, Reza Tourani
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18617
• Pdf URL: https://arxiv.org/pdf/2501.18617
• Copy Paste: [[2501.18617]] DarkMind: Latent Chain-of-Thought Backdoor in Customized LLMs(https://arxiv.org/abs/2501.18617)
• Keywords: security, defense, attack
• Abstract: With the growing demand for personalized AI solutions, customized LLMs have become a preferred choice for businesses and individuals, driving the deployment of millions of AI agents across various platforms, e.g., GPT Store hosts over 3 million customized GPTs. Their popularity is partly driven by advanced reasoning capabilities, such as Chain-of-Thought, which enhance their ability to tackle complex tasks. However, their rapid proliferation introduces new vulnerabilities, particularly in reasoning processes that remain largely unexplored. We introduce DarkMind, a novel backdoor attack that exploits the reasoning capabilities of customized LLMs. Designed to remain latent, DarkMind activates within the reasoning chain to covertly alter the final outcome. Unlike existing attacks, it operates without injecting triggers into user queries, making it a more potent threat. We evaluate DarkMind across eight datasets covering arithmetic, commonsense, and symbolic reasoning domains, using five state-of-the-art LLMs with five distinct trigger implementations. Our results demonstrate DarkMind effectiveness across all scenarios, underscoring its impact. Finally, we explore potential defense mechanisms to mitigate its risks, emphasizing the need for stronger security measures.

Title: Vision Aided Channel Prediction for Vehicular Communications: A Case Study of Received Power Prediction Using RGB Images

• Authors: Xuejian Zhang, Ruisi He, Mi Yang, Zhengyu Zhang, Ziyi Qi, Bo Ai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18618
• Pdf URL: https://arxiv.org/pdf/2501.18618
• Copy Paste: [[2501.18618]] Vision Aided Channel Prediction for Vehicular Communications: A Case Study of Received Power Prediction Using RGB Images(https://arxiv.org/abs/2501.18618)
• Keywords: extraction, segmentation
• Abstract: The communication scenarios and channel characteristics of 6G will be more complex and difficult to characterize. Conventional methods for channel prediction face challenges in achieving an optimal balance between accuracy, practicality, and generalizability. Additionally, they often fail to effectively leverage environmental features. Within the framework of integration communication and artificial intelligence as a pivotal development vision for 6G, it is imperative to achieve intelligent prediction of channel characteristics. Vision-aided methods have been employed in various wireless communication tasks, excluding channel prediction, and have demonstrated enhanced efficiency and performance. In this paper, we propose a vision-aided two-stage model for channel prediction in millimeter wave vehicular communication scenarios, realizing accurate received power prediction utilizing solely RGB images. Firstly, we obtain original images of propagation environment through an RGB camera. Secondly, three typical computer vision methods including object detection, instance segmentation and binary mask are employed for environmental information extraction from original images in stage 1, and prediction of received power based on processed images is implemented in stage 2. Pre-trained YOLOv8 and ResNets are used in stages 1 and 2, respectively, and fine-tuned on datasets. Finally, we conduct five experiments to evaluate the performance of proposed model, demonstrating its feasibility, accuracy and generalization capabilities. The model proposed in this paper offers novel solutions for achieving intelligent channel prediction in vehicular communications.

Title: VLMaterial: Procedural Material Generation with Large Vision-Language Models

• Authors: Beichen Li, Rundi Wu, Armando Solar-Lezama, Changxi Zheng, Liang Shi, Bernd Bickel, Wojciech Matusik
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.18623
• Pdf URL: https://arxiv.org/pdf/2501.18623
• Copy Paste: [[2501.18623]] VLMaterial: Procedural Material Generation with Large Vision-Language Models(https://arxiv.org/abs/2501.18623)
• Keywords: large language model
• Abstract: Procedural materials, represented as functional node graphs, are ubiquitous in computer graphics for photorealistic material appearance design. They allow users to perform intuitive and precise editing to achieve desired visual appearances. However, creating a procedural material given an input image requires professional knowledge and significant effort. In this work, we leverage the ability to convert procedural materials into standard Python programs and fine-tune a large pre-trained vision-language model (VLM) to generate such programs from input images. To enable effective fine-tuning, we also contribute an open-source procedural material dataset and propose to perform program-level augmentation by prompting another pre-trained large language model (LLM). Through extensive evaluation, we show that our method outperforms previous methods on both synthetic and real-world examples.

Title: Membership Inference Attacks Against Vision-Language Models

• Authors: Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, Chun Chen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18624
• Pdf URL: https://arxiv.org/pdf/2501.18624
• Copy Paste: [[2501.18624]] Membership Inference Attacks Against Vision-Language Models(https://arxiv.org/abs/2501.18624)
• Keywords: attack, membership infer, large language model
• Abstract: Vision-Language Models (VLMs), built on pre-trained vision encoders and large language models (LLMs), have shown exceptional multi-modal understanding and dialog capabilities, positioning them as catalysts for the next technological revolution. However, while most VLM research focuses on enhancing multi-modal interaction, the risks of data misuse and leakage have been largely unexplored. This prompts the need for a comprehensive investigation of such risks in VLMs. In this paper, we conduct the first analysis of misuse and leakage detection in VLMs through the lens of membership inference attack (MIA). In specific, we focus on the instruction tuning data of VLMs, which is more likely to contain sensitive or unauthorized information. To address the limitation of existing MIA methods, we introduce a novel approach that infers membership based on a set of samples and their sensitivity to temperature, a unique parameter in VLMs. Based on this, we propose four membership inference methods, each tailored to different levels of background knowledge, ultimately arriving at the most challenging scenario. Our comprehensive evaluations show that these methods can accurately determine membership status, e.g., achieving an AUC greater than 0.8 targeting a small set consisting of only 5 samples on LLaVA.

Title: DUEF-GA: Data Utility and Privacy Evaluation Framework for Graph Anonymization

• Authors: Jordi Casas-Roma
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18625
• Pdf URL: https://arxiv.org/pdf/2501.18625
• Copy Paste: [[2501.18625]] DUEF-GA: Data Utility and Privacy Evaluation Framework for Graph Anonymization(https://arxiv.org/abs/2501.18625)
• Keywords: privacy
• Abstract: Anonymization of graph-based data is a problem which has been widely studied over the last years and several anonymization methods have been developed. Information loss measures have been used to evaluate data utility and information loss in the anonymized graphs. However, there is no consensus about how to evaluate data utility and information loss in privacy-preserving and anonymization scenarios, where the anonymous datasets were perturbed to hinder re-identification processes. Authors use diverse metrics to evaluate data utility and, consequently, it is complex to compare different methods or algorithms in literature. In this paper we propose a framework to evaluate and compare anonymous datasets in a common way, providing an objective score to clearly compare methods and algorithms. Our framework includes metrics based on generic information loss measures, such as average distance or betweenness centrality, and also task-specific information loss measures, such as community detection or information flow. Additionally, we provide some metrics to examine re-identification and risk assessment. We demonstrate that our framework could help researchers and practitioners to select the best parametrization and/or algorithm to reduce information loss and maximize data utility.

Title: The TIP of the Iceberg: Revealing a Hidden Class of Task-In-Prompt Adversarial Attacks on LLMs

• Authors: Sergey Berezin, Reza Farahbakhsh, Noel Crespi
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18626
• Pdf URL: https://arxiv.org/pdf/2501.18626
• Copy Paste: [[2501.18626]] The TIP of the Iceberg: Revealing a Hidden Class of Task-In-Prompt Adversarial Attacks on LLMs(https://arxiv.org/abs/2501.18626)
• Keywords: attack
• Abstract: We present a novel class of jailbreak adversarial attacks on LLMs, termed Task-in-Prompt (TIP) attacks. Our approach embeds sequence-to-sequence tasks (e.g., cipher decoding, riddles, code execution) into the model's prompt to indirectly generate prohibited inputs. To systematically assess the effectiveness of these attacks, we introduce the PHRYGE benchmark. We demonstrate that our techniques successfully circumvent safeguards in six state-of-the-art language models, including GPT-4o and LLaMA 3.2. Our findings highlight critical weaknesses in current LLM safety alignments and underscore the urgent need for more sophisticated defence strategies. Warning: this paper contains examples of unethical inquiries used solely for research purposes.

Title: Indiana Jones: There Are Always Some Useful Ancient Relics

• Authors: Junchen Ding, Jiahao Zhang, Yi Liu, Ziqi Ding, Gelei Deng, Yuekang Li
• Subjects: cs.CR, cs.AI, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.18628
• Pdf URL: https://arxiv.org/pdf/2501.18628
• Copy Paste: [[2501.18628]] Indiana Jones: There Are Always Some Useful Ancient Relics(https://arxiv.org/abs/2501.18628)
• Keywords: security, robust, large language model
• Abstract: This paper introduces Indiana Jones, an innovative approach to jailbreaking Large Language Models (LLMs) by leveraging inter-model dialogues and keyword-driven prompts. Through orchestrating interactions among three specialised LLMs, the method achieves near-perfect success rates in bypassing content safeguards in both white-box and black-box LLMs. The research exposes systemic vulnerabilities within contemporary models, particularly their susceptibility to producing harmful or unethical outputs when guided by ostensibly innocuous prompts framed in historical or contextual contexts. Experimental evaluations highlight the efficacy and adaptability of Indiana Jones, demonstrating its superiority over existing jailbreak methods. These findings emphasise the urgent need for enhanced ethical safeguards and robust security measures in the development of LLMs. Moreover, this work provides a critical foundation for future studies aimed at fortifying LLMs against adversarial exploitation while preserving their utility and flexibility.

Title: The Relationship Between Network Similarity and Transferability of Adversarial Attacks

• Authors: Gerrit Klause, Niklas Bunzel
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18629
• Pdf URL: https://arxiv.org/pdf/2501.18629
• Copy Paste: [[2501.18629]] The Relationship Between Network Similarity and Transferability of Adversarial Attacks(https://arxiv.org/abs/2501.18629)
• Keywords: defense, attack, robust
• Abstract: Neural networks are vulnerable to adversarial attacks, and several defenses have been proposed. Designing a robust network is a challenging task given the wide range of attacks that have been developed. Therefore, we aim to provide insight into the influence of network similarity on the success rate of transferred adversarial attacks. Network designers can then compare their new network with existing ones to estimate its vulnerability. To achieve this, we investigate the complex relationship between network similarity and the success rate of transferred adversarial attacks. We applied the Centered Kernel Alignment (CKA) network similarity score and used various methods to find a correlation between a large number of Convolutional Neural Networks (CNNs) and adversarial attacks. Network similarity was found to be moderate across different CNN architectures, with more complex models such as DenseNet showing lower similarity scores due to their architectural complexity. Layer similarity was highest for consistent, basic layers such as DataParallel, Dropout and Conv2d, while specialized layers showed greater variability. Adversarial attack success rates were generally consistent for non-transferred attacks, but varied significantly for some transferred attacks, with complex networks being more vulnerable. We found that a DecisionTreeRegressor can predict the success rate of transferred attacks for all black-box and Carlini & Wagner attacks with an accuracy of over 90%, suggesting that predictive models may be viable under certain conditions. However, the variability of results across different data subsets underscores the complexity of these relationships and suggests that further research is needed to generalize these findings across different attack scenarios and network architectures.

Title: Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare

• Authors: Hang Zhang, Qian Lou, Yanshan Wang
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18632
• Pdf URL: https://arxiv.org/pdf/2501.18632
• Copy Paste: [[2501.18632]] Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare(https://arxiv.org/abs/2501.18632)
• Keywords: attack, large language model
• Abstract: Large language models (LLMs) are increasingly utilized in healthcare applications. However, their deployment in clinical practice raises significant safety concerns, including the potential spread of harmful information. This study systematically assesses the vulnerabilities of six LLMs to three advanced black-box jailbreaking techniques within medical contexts. To quantify the effectiveness of these techniques, we propose an automated and domain-adapted agentic evaluation pipeline. Experiment results indicate that leading commercial and open-source LLMs are highly vulnerable to medical jailbreaking attacks. To bolster model safety and reliability, we further investigate the effectiveness of Continual Fine-Tuning (CFT) in defending against medical adversarial attacks. Our findings underscore the necessity for evolving attack methods evaluation, domain-specific safety alignment, and LLM safety-utility balancing. This research offers actionable insights for advancing the safety and reliability of AI clinicians, contributing to ethical and effective AI deployment in healthcare.

Title: SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model

• Authors: Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Jason Zhaoxin Fan, Bo Tang, Shichao Song, Mengwei Wang, Jiawei Yang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18636
• Pdf URL: https://arxiv.org/pdf/2501.18636
• Copy Paste: [[2501.18636]] SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model(https://arxiv.org/abs/2501.18636)
• Keywords: security, attack, large language model
• Abstract: The indexing-retrieval-generation paradigm of retrieval-augmented generation (RAG) has been highly successful in solving knowledge-intensive tasks by integrating external knowledge into large language models (LLMs). However, the incorporation of external and unverified knowledge increases the vulnerability of LLMs because attackers can perform attack tasks by manipulating knowledge. In this paper, we introduce a benchmark named SafeRAG designed to evaluate the RAG security. First, we classify attack tasks into silver noise, inter-context conflict, soft ad, and white Denial-of-Service. Next, we construct RAG security evaluation dataset (i.e., SafeRAG dataset) primarily manually for each task. We then utilize the SafeRAG dataset to simulate various attack scenarios that RAG may encounter. Experiments conducted on 14 representative RAG components demonstrate that RAG exhibits significant vulnerability to all attack tasks and even the most apparent attack task can easily bypass existing retrievers, filters, or advanced LLMs, resulting in the degradation of RAG service quality. Code is available at: this https URL.

Title: Machine learning of microstructure--property relationships in materials with robust features from foundational vision transformers

• Authors: Sheila E. Whitman, Marat I. Latypov
• Subjects: cs.CV, cond-mat.mtrl-sci, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18637
• Pdf URL: https://arxiv.org/pdf/2501.18637
• Copy Paste: [[2501.18637]] Machine learning of microstructure--property relationships in materials with robust features from foundational vision transformers(https://arxiv.org/abs/2501.18637)
• Keywords: robust, extraction, transformer
• Abstract: Machine learning of microstructure--property relationships from data is an emerging approach in computational materials science. Most existing machine learning efforts focus on the development of task-specific models for each microstructure--property relationship. We propose utilizing pre-trained foundational vision transformers for the extraction of task-agnostic microstructure features and subsequent light-weight machine learning of a microstructure-dependent property. We demonstrate our approach with pre-trained state-of-the-art vision transformers (CLIP, DINOV2, SAM) in two case studies on machine-learning: (i) elastic modulus of two-phase microstructures based on simulations data; and (ii) Vicker's hardness of Ni-base and Co-base superalloys based on experimental data published in literature. Our results show the potential of foundational vision transformers for robust microstructure representation and efficient machine learning of microstructure--property relationships without the need for expensive task-specific training or fine-tuning of bespoke deep learning models.

Title: Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderation

• Authors: Daniel Schwartz, Dmitriy Bespalov, Zhe Wang, Ninad Kulkarni, Yanjun Qi
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18638
• Pdf URL: https://arxiv.org/pdf/2501.18638
• Copy Paste: [[2501.18638]] Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderation(https://arxiv.org/abs/2501.18638)
• Keywords: attack, steal
• Abstract: We present a modular pipeline that automates the generation of stealthy jailbreak prompts derived from high-level content policies, enhancing LLM content moderation. First, we address query inefficiency and jailbreak strength by developing Graph of Attacks with Pruning (GAP), a method that utilizes strategies from prior jailbreaks, resulting in 92% attack success rate on GPT-3.5 using only 54% of the queries of the prior algorithm. Second, we address the cold-start issue by automatically generating seed prompts from the high-level policy using LLMs. Finally, we demonstrate the utility of these generated jailbreak prompts of improving content moderation by fine-tuning PromptGuard, a model trained to detect jailbreaks, increasing its accuracy on the Toxic-Chat dataset from 5.1% to 93.89%.

Title: Divergent Emotional Patterns in Disinformation on Social Media? An Analysis of Tweets and TikToks about the DANA in Valencia

• Authors: Iván Arcos, Paolo Rosso, Ramón Salaverría
• Subjects: cs.CL, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.18640
• Pdf URL: https://arxiv.org/pdf/2501.18640
• Copy Paste: [[2501.18640]] Divergent Emotional Patterns in Disinformation on Social Media? An Analysis of Tweets and TikToks about the DANA in Valencia(https://arxiv.org/abs/2501.18640)
• Keywords: large language model
• Abstract: This study investigates the dissemination of disinformation on social media platforms during the DANA event (DANA is a Spanish acronym for Depresion Aislada en Niveles Altos, translating to high-altitude isolated depression) that resulted in extremely heavy rainfall and devastating floods in Valencia, Spain, on October 29, 2024. We created a novel dataset of 650 TikTok and X posts, which was manually annotated to differentiate between disinformation and trustworthy content. Additionally, a Few-Shot annotation approach with GPT-4o achieved substantial agreement (Cohen's kappa of 0.684) with manual labels. Emotion analysis revealed that disinformation on X is mainly associated with increased sadness and fear, while on TikTok, it correlates with higher levels of anger and disgust. Linguistic analysis using the LIWC dictionary showed that trustworthy content utilizes more articulate and factual language, whereas disinformation employs negations, perceptual words, and personal anecdotes to appear credible. Audio analysis of TikTok posts highlighted distinct patterns: trustworthy audios featured brighter tones and robotic or monotone narration, promoting clarity and credibility, while disinformation audios leveraged tonal variation, emotional depth, and manipulative musical elements to amplify engagement. In detection models, SVM+TF-IDF achieved the highest F1-Score, excelling with limited data. Incorporating audio features into roberta-large-bne improved both Accuracy and F1-Score, surpassing its text-only counterpart and SVM in Accuracy. GPT-4o Few-Shot also performed well, showcasing the potential of large language models for automated disinformation detection. These findings demonstrate the importance of leveraging both textual and audio features for improved disinformation detection on multimodal platforms like TikTok.

Title: Image Velocimetry using Direct Displacement Field estimation with Neural Networks for Fluids

• Authors: Efraín Magaña, Francisco Sahli Costabal, Wernher Brevis
• Subjects: cs.CV, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2501.18641
• Pdf URL: https://arxiv.org/pdf/2501.18641
• Copy Paste: [[2501.18641]] Image Velocimetry using Direct Displacement Field estimation with Neural Networks for Fluids(https://arxiv.org/abs/2501.18641)
• Keywords: robust
• Abstract: An important tool for experimental fluids mechanics research is Particle Image Velocimetry (PIV). Several robust methodologies have been proposed to perform the estimation of velocity field from the images, however, alternative methods are still needed to increase the spatial resolution of the results. This work presents a novel approach for estimating fluid flow fields using neural networks and the optical flow equation to predict displacement vectors between sequential images. The result is a continuous representation of the displacement, that can be evaluated on the full spatial resolution of the image. The methodology was validated on synthetic and experimental images. Accurate results were obtained in terms of the estimation of instantaneous velocity fields, and of the determined time average turbulence quantities and power spectral density. The methodology proposed differs of previous attempts of using machine learning for this task: it does not require any previous training, and could be directly used in any pair of images.

Title: DebiasPI: Inference-time Debiasing by Prompt Iteration of a Text-to-Image Generative Model

• Authors: Sarah Bonna, Yu-Cheng Huang, Ekaterina Novozhilova, Sejin Paik, Zhengyang Shan, Michelle Yilin Feng, Ge Gao, Yonish Tayal, Rushil Kulkarni, Jialin Yu, Nupur Divekar, Deepti Ghadiyaram, Derry Wijaya, Margrit Betke
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.18642
• Pdf URL: https://arxiv.org/pdf/2501.18642
• Copy Paste: [[2501.18642]] DebiasPI: Inference-time Debiasing by Prompt Iteration of a Text-to-Image Generative Model(https://arxiv.org/abs/2501.18642)
• Keywords: generative
• Abstract: Ethical intervention prompting has emerged as a tool to counter demographic biases of text-to-image generative AI models. Existing solutions either require to retrain the model or struggle to generate images that reflect desired distributions on gender and race. We propose an inference-time process called DebiasPI for Debiasing-by-Prompt-Iteration that provides prompt intervention by enabling the user to control the distributions of individuals' demographic attributes in image generation. DebiasPI keeps track of which attributes have been generated either by probing the internal state of the model or by using external attribute classifiers. Its control loop guides the text-to-image model to select not yet sufficiently represented attributes, With DebiasPI, we were able to create images with equal representations of race and gender that visualize challenging concepts of news headlines. We also experimented with the attributes age, body type, profession, and skin tone, and measured how attributes change when our intervention prompt targets the distribution of an unrelated attribute type. We found, for example, if the text-to-image model is asked to balance racial representation, gender representation improves but the skin tone becomes less diverse. Attempts to cover a wide range of skin colors with various intervention prompts showed that the model struggles to generate the palest skin tones. We conducted various ablation studies, in which we removed DebiasPI's attribute control, that reveal the model's propensity to generate young, male characters. It sometimes visualized career success by generating two-panel images with a pre-success dark-skinned person becoming light-skinned with success, or switching gender from pre-success female to post-success male, thus further motivating ethical intervention prompting with DebiasPI.

Title: 3D Reconstruction of Shoes for Augmented Reality

• Authors: Pratik Shrestha, Sujan Kapali, Swikar Gautam, Vishal Pokharel, Santosh Giri
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.18643
• Pdf URL: https://arxiv.org/pdf/2501.18643
• Copy Paste: [[2501.18643]] 3D Reconstruction of Shoes for Augmented Reality(https://arxiv.org/abs/2501.18643)
• Keywords: segmentation
• Abstract: This paper introduces a mobile-based solution that enhances online shoe shopping through 3D modeling and Augmented Reality (AR), leveraging the efficiency of 3D Gaussian Splatting. Addressing the limitations of static 2D images, the framework generates realistic 3D shoe models from 2D images, achieving an average Peak Signal-to-Noise Ratio (PSNR) of 0.32, and enables immersive AR interactions via smartphones. A custom shoe segmentation dataset of 3120 images was created, with the best-performing segmentation model achieving an Intersection over Union (IoU) score of 0.95. This paper demonstrates the potential of 3D modeling and AR to revolutionize online shopping by offering realistic virtual interactions, with applicability across broader fashion categories.

Title: Prompt-oriented Output of Culture-Specific Items in Translated African Poetry by Large Language Model: An Initial Multi-layered Tabular Review

• Authors: Adeyola Opaluwah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18644
• Pdf URL: https://arxiv.org/pdf/2501.18644
• Copy Paste: [[2501.18644]] Prompt-oriented Output of Culture-Specific Items in Translated African Poetry by Large Language Model: An Initial Multi-layered Tabular Review(https://arxiv.org/abs/2501.18644)
• Keywords: transformer, generative, large language model
• Abstract: This paper examines the output of cultural items generated by Chat Generative PreTrained Transformer Pro in response to three structured prompts to translate three anthologies of African poetry. The first prompt was broad, the second focused on poetic structure, and the third prompt emphasized cultural specificity. To support this analysis, four comparative tables were created. The first table presents the results of the cultural items produced after the three prompts, the second categorizes these outputs based on Aixela framework of Proper nouns and Common expressions, the third table summarizes the cultural items generated by human translators, a custom translation engine, and a Large Language Model. The final table outlines the strategies employed by Chat Generative PreTrained Transformer Pro following the culture specific prompt. Compared to the outputs of cultural items from reference human translation and the custom translation engine in prior studies the findings indicate that the culture oriented prompts used with Chat Generative PreTrained Transformer Pro did not yield significant enhancements of cultural items during the translation of African poetry from English to French. Among the fifty four cultural items, the human translation produced thirty three cultural items in repetition, the custom translation engine generated Thirty eight cultural items in repetition while Chat Generative PreTrained Transformer Pro produced forty one cultural items in repetition. The untranslated cultural items revealed inconsistencies in Large language models approach to translating cultural items in African poetry from English to French.

Title: Layered Chain-of-Thought Prompting for Multi-Agent LLM Systems: A Comprehensive Approach to Explainable Large Language Models

• Authors: Manish Sanwal
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.18645
• Pdf URL: https://arxiv.org/pdf/2501.18645
• Copy Paste: [[2501.18645]] Layered Chain-of-Thought Prompting for Multi-Agent LLM Systems: A Comprehensive Approach to Explainable Large Language Models(https://arxiv.org/abs/2501.18645)
• Keywords: explainability, large language model
• Abstract: Large Language Models (LLMs) leverage chain-of-thought (CoT) prompting to provide step-by-step rationales, improving performance on complex tasks. Despite its benefits, vanilla CoT often fails to fully verify intermediate inferences and can produce misleading explanations. In this work, we propose Layered Chain-of-Thought (Layered-CoT) Prompting, a novel framework that systematically segments the reasoning process into multiple layers, each subjected to external checks and optional user feedback. We expand on the key concepts, present three scenarios -- medical triage, financial risk assessment, and agile engineering -- and demonstrate how Layered-CoT surpasses vanilla CoT in terms of transparency, correctness, and user engagement. By integrating references from recent arXiv papers on interactive explainability, multi-agent frameworks, and agent-based collaboration, we illustrate how Layered-CoT paves the way for more reliable and grounded explanations in high-stakes domains.

Title: Image, Text, and Speech Data Augmentation using Multimodal LLMs for Deep Learning: A Survey

• Authors: Ranjan Sapkota, Shaina Raza, Maged Shoman, Achyut Paudel, Manoj Karkee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18648
• Pdf URL: https://arxiv.org/pdf/2501.18648
• Copy Paste: [[2501.18648]] Image, Text, and Speech Data Augmentation using Multimodal LLMs for Deep Learning: A Survey(https://arxiv.org/abs/2501.18648)
• Keywords: large language model
• Abstract: In the past five years, research has shifted from traditional Machine Learning (ML) and Deep Learning (DL) approaches to leveraging Large Language Models (LLMs) , including multimodality, for data augmentation to enhance generalization, and combat overfitting in training deep convolutional neural networks. However, while existing surveys predominantly focus on ML and DL techniques or limited modalities (text or images), a gap remains in addressing the latest advancements and multi-modal applications of LLM-based methods. This survey fills that gap by exploring recent literature utilizing multimodal LLMs to augment image, text, and audio data, offering a comprehensive understanding of these processes. We outlined various methods employed in the LLM-based image, text and speech augmentation, and discussed the limitations identified in current approaches. Additionally, we identified potential solutions to these limitations from the literature to enhance the efficacy of data augmentation practices using multimodal LLMs. This survey serves as a foundation for future research, aiming to refine and expand the use of multimodal LLMs in enhancing dataset quality and diversity for deep learning applications. (Surveyed Paper GitHub Repo: this https URL. Keywords: LLM data augmentation, LLM text data augmentation, LLM image data augmentation, LLM speech data augmentation, audio augmentation, voice augmentation, chatGPT for data augmentation, DeepSeek R1 text data augmentation, DeepSeek R1 image augmentation, Image Augmentation using LLM, Text Augmentation using LLM, LLM data augmentation for deep learning applications)

Title: Fake News Detection After LLM Laundering: Measurement and Explanation

• Authors: Rupak Kumar Das, Jonathan Dodge
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18649
• Pdf URL: https://arxiv.org/pdf/2501.18649
• Copy Paste: [[2501.18649]] Fake News Detection After LLM Laundering: Measurement and Explanation(https://arxiv.org/abs/2501.18649)
• Keywords: large language model
• Abstract: With their advanced capabilities, Large Language Models (LLMs) can generate highly convincing and contextually relevant fake news, which can contribute to disseminating misinformation. Though there is much research on fake news detection for human-written text, the field of detecting LLM-generated fake news is still under-explored. This research measures the efficacy of detectors in identifying LLM-paraphrased fake news, in particular, determining whether adding a paraphrase step in the detection pipeline helps or impedes detection. This study contributes: (1) Detectors struggle to detect LLM-paraphrased fake news more than human-written text, (2) We find which models excel at which tasks (evading detection, paraphrasing to evade detection, and paraphrasing for semantic similarity). (3) Via LIME explanations, we discovered a possible reason for detection failures: sentiment shift. (4) We discover a worrisome trend for paraphrase quality measurement: samples that exhibit sentiment shift despite a high BERTSCORE. (5) We provide a pair of datasets augmenting existing datasets with paraphrase outputs and scores. The dataset is available on GitHub

Title: SAFL: Structure-Aware Personalized Federated Learning via Client-Specific Clustering and SCSI-Guided Model Pruning

• Authors: Nan Li, Xiaolu Wang, Xiao Du, Puyu Cai, Ting Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18659
• Pdf URL: https://arxiv.org/pdf/2501.18659
• Copy Paste: [[2501.18659]] SAFL: Structure-Aware Personalized Federated Learning via Client-Specific Clustering and SCSI-Guided Model Pruning(https://arxiv.org/abs/2501.18659)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) enables clients to collaboratively train machine learning models without sharing local data, preserving privacy in diverse environments. While traditional FL approaches preserve privacy, they often struggle with high computational and communication overhead. To address these issues, model pruning is introduced as a strategy to streamline computations. However, existing pruning methods, when applied solely based on local data, often produce sub-models that inadequately reflect clients' specific tasks due to data insufficiency. To overcome these challenges, this paper introduces SAFL (Structure-Aware Federated Learning), a novel framework that enhances personalized federated learning through client-specific clustering and Similar Client Structure Information (SCSI)-guided model pruning. SAFL employs a two-stage process: initially, it groups clients based on data similarities and uses aggregated pruning criteria to guide the pruning process, facilitating the identification of optimal sub-models. Subsequently, clients train these pruned models and engage in server-based aggregation, ensuring tailored and efficient models for each client. This method significantly reduces computational overhead while improving inference accuracy. Extensive experiments demonstrate that SAFL markedly diminishes model size and improves performance, making it highly effective in federated environments characterized by heterogeneous data.

Title: Joint Optimization of Prompt Security and System Performance in Edge-Cloud LLM Systems

• Authors: Haiyang Huang, Tianhui Meng, Weijia Jia
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18663
• Pdf URL: https://arxiv.org/pdf/2501.18663
• Copy Paste: [[2501.18663]] Joint Optimization of Prompt Security and System Performance in Edge-Cloud LLM Systems(https://arxiv.org/abs/2501.18663)
• Keywords: security, privacy, attack, large language model
• Abstract: Large language models (LLMs) have significantly facilitated human life, and prompt engineering has improved the efficiency of these models. However, recent years have witnessed a rise in prompt engineering-empowered attacks, leading to issues such as privacy leaks, increased latency, and system resource wastage. Though safety fine-tuning based methods with Reinforcement Learning from Human Feedback (RLHF) are proposed to align the LLMs, existing security mechanisms fail to cope with fickle prompt attacks, highlighting the necessity of performing security detection on prompts. In this paper, we jointly consider prompt security, service latency, and system resource optimization in Edge-Cloud LLM (EC-LLM) systems under various prompt attacks. To enhance prompt security, a vector-database-enabled lightweight attack detector is proposed. We formalize the problem of joint prompt detection, latency, and resource optimization into a multi-stage dynamic Bayesian game model. The equilibrium strategy is determined by predicting the number of malicious tasks and updating beliefs at each stage through Bayesian updates. The proposed scheme is evaluated on a real implemented EC-LLM system, and the results demonstrate that our approach offers enhanced security, reduces the service latency for benign users, and decreases system resource consumption compared to state-of-the-art algorithms.

Title: BARNN: A Bayesian Autoregressive and Recurrent Neural Network

• Authors: Dario Coscia, Max Welling, Nicola Demo, Gianluigi Rozza
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18665
• Pdf URL: https://arxiv.org/pdf/2501.18665
• Copy Paste: [[2501.18665]] BARNN: A Bayesian Autoregressive and Recurrent Neural Network(https://arxiv.org/abs/2501.18665)
• Keywords: large language model
• Abstract: Autoregressive and recurrent networks have achieved remarkable progress across various fields, from weather forecasting to molecular generation and Large Language Models. Despite their strong predictive capabilities, these models lack a rigorous framework for addressing uncertainty, which is key in scientific applications such as PDE solving, molecular generation and Machine Learning Force Fields. To address this shortcoming we present BARNN: a variational Bayesian Autoregressive and Recurrent Neural Network. BARNNs aim to provide a principled way to turn any autoregressive or recurrent model into its Bayesian version. BARNN is based on the variational dropout method, allowing to apply it to large recurrent neural networks as well. We also introduce a temporal version of the "Variational Mixtures of Posteriors" prior (tVAMP-prior) to make Bayesian inference efficient and well-calibrated. Extensive experiments on PDE modelling and molecular generation demonstrate that BARNN not only achieves comparable or superior accuracy compared to existing methods, but also excels in uncertainty quantification and modelling long-range dependencies.

Title: Structure Development in List-Sorting Transformers

• Authors: Einar Urdshals, Jasmina Urdshals
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2501.18666
• Pdf URL: https://arxiv.org/pdf/2501.18666
• Copy Paste: [[2501.18666]] Structure Development in List-Sorting Transformers(https://arxiv.org/abs/2501.18666)
• Keywords: transformer
• Abstract: We study how a one-layer attention-only transformer develops relevant structures while learning to sort lists of numbers. At the end of training, the model organizes its attention heads in two main modes that we refer to as vocabulary-splitting and copy-suppression. Both represent simpler modes than having multiple heads handle overlapping ranges of numbers. Interestingly, vocabulary-splitting is present regardless of whether we use weight decay, a common regularization technique thought to drive simplification, supporting the thesis that neural networks naturally prefer simpler solutions. We relate copy-suppression to a mechanism in GPT-2 and investigate its functional role in our model. Guided by insights from a developmental analysis of the model, we identify features in the training data that drive the model's final acquired solution. This provides a concrete example of how the training data shape the internal organization of transformers, paving the way for future studies that could help us better understand how LLMs develop their internal structures.

Title: The Pitfalls of "Security by Obscurity" And What They Mean for Transparent AI

• Authors: Peter Hall, Olivia Mundahl, Sunoo Park
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.18669
• Pdf URL: https://arxiv.org/pdf/2501.18669
• Copy Paste: [[2501.18669]] The Pitfalls of "Security by Obscurity" And What They Mean for Transparent AI(https://arxiv.org/abs/2501.18669)
• Keywords: security, protect, attack
• Abstract: Calls for transparency in AI systems are growing in number and urgency from diverse stakeholders ranging from regulators to researchers to users (with a comparative absence of companies developing AI). Notions of transparency for AI abound, each addressing distinct interests and concerns. In computer security, transparency is likewise regarded as a key concept. The security community has for decades pushed back against so-called security by obscurity -- the idea that hiding how a system works protects it from attack -- against significant pressure from industry and other stakeholders. Over the decades, in a community process that is imperfect and ongoing, security researchers and practitioners have gradually built up some norms and practices around how to balance transparency interests with possible negative side effects. This paper asks: What insights can the AI community take from the security community's experience with transparency? We identify three key themes in the security community's perspective on the benefits of transparency and their approach to balancing transparency against countervailing interests. For each, we investigate parallels and insights relevant to transparency in AI. We then provide a case study discussion on how transparency has shaped the research subfield of anonymization. Finally, shifting our focus from similarities to differences, we highlight key transparency issues where modern AI systems present challenges different from other kinds of security-critical systems, raising interesting open questions for the security and AI communities alike.

Title: Unpaired Translation of Point Clouds for Modeling Detector Response

• Authors: Mingyang Li, Michelle Kuchera, Raghuram Ramanujan, Adam Anthony, Curtis Hunt, Yassid Ayyad
• Subjects: cs.CV, cs.LG, nucl-ex
• Abstract URL: https://arxiv.org/abs/2501.18674
• Pdf URL: https://arxiv.org/pdf/2501.18674
• Copy Paste: [[2501.18674]] Unpaired Translation of Point Clouds for Modeling Detector Response(https://arxiv.org/abs/2501.18674)
• Keywords: diffusion
• Abstract: Modeling detector response is a key challenge in time projection chambers. We cast this problem as an unpaired point cloud translation task, between data collected from simulations and from experimental runs. Effective translation can assist with both noise rejection and the construction of high-fidelity simulators. Building on recent work in diffusion probabilistic models, we present a novel framework for performing this mapping. We demonstrate the success of our approach in both synthetic domains and in data sourced from the Active-Target Time Projection Chamber.

Title: Regularized second-order optimization of tensor-network Born machines

• Authors: Matan Ben-Dov, Jing Chen
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2501.18691
• Pdf URL: https://arxiv.org/pdf/2501.18691
• Copy Paste: [[2501.18691]] Regularized second-order optimization of tensor-network Born machines(https://arxiv.org/abs/2501.18691)
• Keywords: generative
• Abstract: Tensor-network Born machines (TNBMs) are quantum-inspired generative models for learning data distributions. Using tensor-network contraction and optimization techniques, the model learns an efficient representation of the target distribution, capable of capturing complex correlations with a compact parameterization. Despite their promise, the optimization of TNBMs presents several challenges. A key bottleneck of TNBMs is the logarithmic nature of the loss function that is commonly used for this problem. The single-tensor logarithmic optimization problem cannot be solved analytically, necessitating an iterative approach that slows down convergence and increases the risk of getting trapped in one of many non-optimal local minima. In this paper, we present an improved second-order optimization technique for TNBM training, which significantly enhances convergence rates and the quality of the optimized model. Our method employs a modified Newton's method on the manifold of normalized states, incorporating regularization of the loss landscape to mitigate local minima issues. We demonstrate the effectiveness of our approach by training a one-dimensional matrix product state (MPS) on both discrete and continuous datasets, showcasing its advantages in terms of stability, efficiency, and generalization.

Title: Invisible Traces: Using Hybrid Fingerprinting to identify underlying LLMs in GenAI Apps

• Authors: Devansh Bhardwaj, Naman Mishra
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18712
• Pdf URL: https://arxiv.org/pdf/2501.18712
• Copy Paste: [[2501.18712]] Invisible Traces: Using Hybrid Fingerprinting to identify underlying LLMs in GenAI Apps(https://arxiv.org/abs/2501.18712)
• Keywords: security, robust, large language model
• Abstract: Fingerprinting refers to the process of identifying underlying Machine Learning (ML) models of AI Systemts, such as Large Language Models (LLMs), by analyzing their unique characteristics or patterns, much like a human fingerprint. The fingerprinting of Large Language Models (LLMs) has become essential for ensuring the security and transparency of AI-integrated applications. While existing methods primarily rely on access to direct interactions with the application to infer model identity, they often fail in real-world scenarios involving multi-agent systems, frequent model updates, and restricted access to model internals. In this paper, we introduce a novel fingerprinting framework designed to address these challenges by integrating static and dynamic fingerprinting techniques. Our approach identifies architectural features and behavioral traits, enabling accurate and robust fingerprinting of LLMs in dynamic environments. We also highlight new threat scenarios where traditional fingerprinting methods are ineffective, bridging the gap between theoretical techniques and practical application. To validate our framework, we present an extensive evaluation setup that simulates real-world conditions and demonstrate the effectiveness of our methods in identifying and monitoring LLMs in Gen-AI applications. Our results highlight the framework's adaptability to diverse and evolving deployment contexts.

Title: Full-Head Segmentation of MRI with Abnormal Brain Anatomy: Model and Data Release

• Authors: Andrew M Birnbaum, Adam Buchwald, Peter Turkeltaub, Adam Jacks, Yu Huang, Abhisheck Datta, Lucas C Parra, Lukas A Hirsch
• Subjects: cs.CV, cs.LG, eess.IV, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2501.18716
• Pdf URL: https://arxiv.org/pdf/2501.18716
• Copy Paste: [[2501.18716]] Full-Head Segmentation of MRI with Abnormal Brain Anatomy: Model and Data Release(https://arxiv.org/abs/2501.18716)
• Keywords: robust, segmentation
• Abstract: The goal of this work was to develop a deep network for whole-head segmentation, including clinical MRIs with abnormal anatomy, and compile the first public benchmark dataset for this purpose. We collected 91 MRIs with volumetric segmentation labels for a diverse set of human subjects (4 normal, 32 traumatic brain injuries, and 57 strokes). These clinical cases are characterized by extended cerebrospinal fluid (CSF) in regions normally containing the brain. Training labels were generated by manually correcting initial automated segmentations for skin/scalp, skull, CSF, gray matter, white matter, air cavity, and extracephalic air. We developed a MultiAxial network consisting of three 2D U-Net models that operate independently in sagittal, axial, and coronal planes and are then combined to produce a single 3D segmentation. The MultiAxial network achieved test-set Dice scores of 0.88 (median plus-minus 0.04). For brain tissue, it significantly outperforms existing brain segmentation methods (MultiAxial: 0.898 plus-minus 0.041, SynthSeg: 0.758 plus-minus 0.054, BrainChop: 0.757 plus-minus 0.125). The MultiAxial network gains in robustness by avoiding the need for coregistration with an atlas. It performed well in regions with abnormal anatomy and on images that have been de-identified. It enables more robust current flow modeling when incorporated into ROAST, a widely-used modeling toolbox for transcranial electric stimulation. We are releasing a state-of-the-art model for whole-head MRI segmentation, along with a dataset of 61 clinical MRIs and training labels, including non-brain structures. Together, the model and data may serve as a benchmark for future efforts.

Title: Zero-shot Large Language Models for Long Clinical Text Summarization with Temporal Reasoning

• Authors: Maya Kruse, Shiyue Hu, Nicholas Derby, Yifu Wu, Samantha Stonbraker, Bingsheng Yao, Dakuo Wang, Elizabeth Goldberg, Yanjun Gao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18724
• Pdf URL: https://arxiv.org/pdf/2501.18724
• Copy Paste: [[2501.18724]] Zero-shot Large Language Models for Long Clinical Text Summarization with Temporal Reasoning(https://arxiv.org/abs/2501.18724)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have shown potential for transforming data processing in healthcare, particularly in understanding complex clinical narratives. This study evaluates the efficacy of zero-shot LLMs in summarizing long clinical texts that require temporal reasoning, a critical aspect for comprehensively capturing patient histories and treatment trajectories. We applied a series of advanced zero-shot LLMs to extensive clinical documents, assessing their ability to integrate and accurately reflect temporal dynamics without prior task-specific training. While the models efficiently identified key temporal events, they struggled with chronological coherence over prolonged narratives. The evaluation, combining quantitative and qualitative methods, highlights the strengths and limitations of zero-shot LLMs in clinical text summarization. The results suggest that while promising, zero-shot LLMs require further refinement to effectively support clinical decision-making processes, underscoring the need for enhanced model training approaches that better capture the nuances of temporal information in long context medical documents.

Title: Strong and Controllable 3D Motion Generation

• Authors: Canxuan Gang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18726
• Pdf URL: https://arxiv.org/pdf/2501.18726
• Copy Paste: [[2501.18726]] Strong and Controllable 3D Motion Generation(https://arxiv.org/abs/2501.18726)
• Keywords: diffusion, transformer, generative
• Abstract: Human motion generation is a significant pursuit in generative computer vision with widespread applications in film-making, video games, AR/VR, and human-robot interaction. Current methods mainly utilize either diffusion-based generative models or autoregressive models for text-to-motion generation. However, they face two significant challenges: (1) The generation process is time-consuming, posing a major obstacle for real-time applications such as gaming, robot manipulation, and other online settings. (2) These methods typically learn a relative motion representation guided by text, making it difficult to generate motion sequences with precise joint-level control. These challenges significantly hinder progress and limit the real-world application of human motion generation techniques. To address this gap, we propose a simple yet effective architecture consisting of two key components. Firstly, we aim to improve hardware efficiency and computational complexity in transformer-based diffusion models for human motion generation. By customizing flash linear attention, we can optimize these models specifically for generating human motion efficiently. Furthermore, we will customize the consistency model in the motion latent space to further accelerate motion generation. Secondly, we introduce Motion ControlNet, which enables more precise joint-level control of human motion compared to previous text-to-motion generation methods. These contributions represent a significant advancement for text-to-motion generation, bringing it closer to real-world applications.

Title: Exploring Audio Editing Features as User-Centric Privacy Defenses Against Emotion Inference Attacks

• Authors: Mohd. Farhan Israk Soumik, W.K.M. Mithsara, Abdur R. Shahid, Ahmed Imteaj
• Subjects: cs.CR, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.18727
• Pdf URL: https://arxiv.org/pdf/2501.18727
• Copy Paste: [[2501.18727]] Exploring Audio Editing Features as User-Centric Privacy Defenses Against Emotion Inference Attacks(https://arxiv.org/abs/2501.18727)
• Keywords: security, privacy, protect, defense, attack, large language model
• Abstract: The rapid proliferation of speech-enabled technologies, including virtual assistants, video conferencing platforms, and wearable devices, has raised significant privacy concerns, particularly regarding the inference of sensitive emotional information from audio data. Existing privacy-preserving methods often compromise usability and security, limiting their adoption in practical scenarios. This paper introduces a novel, user-centric approach that leverages familiar audio editing techniques, specifically pitch and tempo manipulation, to protect emotional privacy without sacrificing usability. By analyzing popular audio editing applications on Android and iOS platforms, we identified these features as both widely available and usable. We rigorously evaluated their effectiveness against a threat model, considering adversarial attacks from diverse sources, including Deep Neural Networks (DNNs), Large Language Models (LLMs), and and reversibility testing. Our experiments, conducted on three distinct datasets, demonstrate that pitch and tempo manipulation effectively obfuscates emotional data. Additionally, we explore the design principles for lightweight, on-device implementation to ensure broad applicability across various devices and platforms.

Title: Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques

• Authors: Anthony Mendil, Felix Putze
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18729
• Pdf URL: https://arxiv.org/pdf/2501.18729
• Copy Paste: [[2501.18729]] Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques(https://arxiv.org/abs/2501.18729)
• Keywords: diffusion, transformer
• Abstract: Attribute manipulation deals with the problem of changing individual attributes of a data point or a time series, while leaving all other aspects unaffected. This work focuses on the domain of human motion, more precisely karate movement patterns. To the best of our knowledge, it presents the first success at manipulating attributes of human motion data. One of the key requirements for achieving attribute manipulation on human motion is a suitable pose representation. Therefore, we design a novel rotation-based pose representation that enables the disentanglement of the human skeleton and the motion trajectory, while still allowing an accurate reconstruction of the original anatomy. The core idea of the manipulation approach is to use a transformer encoder for discovering high-level semantics, and a diffusion probabilistic model for modeling the remaining stochastic variations. We show that the embedding space obtained from the transformer encoder is semantically meaningful and linear. This enables the manipulation of high-level attributes, by discovering their linear direction of change in the semantic embedding space and moving the embedding along said direction. The code and data are available at this https URL.

Title: Evaluating Spoken Language as a Biomarker for Automated Screening of Cognitive Impairment

• Authors: Maria R. Lima, Alexander Capstick, Fatemeh Geranmayeh, Ramin Nilforooshan, Maja Matarić, Ravi Vaidyanathan, Payam Barnaghi
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18731
• Pdf URL: https://arxiv.org/pdf/2501.18731
• Copy Paste: [[2501.18731]] Evaluating Spoken Language as a Biomarker for Automated Screening of Cognitive Impairment(https://arxiv.org/abs/2501.18731)
• Keywords: interpretability
• Abstract: Timely and accurate assessment of cognitive impairment is a major unmet need in populations at risk. Alterations in speech and language can be early predictors of Alzheimer's disease and related dementias (ADRD) before clinical signs of neurodegeneration. Voice biomarkers offer a scalable and non-invasive solution for automated screening. However, the clinical applicability of machine learning (ML) remains limited by challenges in generalisability, interpretability, and access to patient data to train clinically applicable predictive models. Using DementiaBank recordings (N=291, 64% female), we evaluated ML techniques for ADRD screening and severity prediction from spoken language. We validated model generalisability with pilot data collected in-residence from older adults (N=22, 59% female). Risk stratification and linguistic feature importance analysis enhanced the interpretability and clinical utility of predictions. For ADRD classification, a Random Forest applied to lexical features achieved a mean sensitivity of 69.4% (95% confidence interval (CI) = 66.4-72.5) and specificity of 83.3% (78.0-88.7). On real-world pilot data, this model achieved a mean sensitivity of 70.0% (58.0-82.0) and specificity of 52.5% (39.3-65.7). For severity prediction using Mini-Mental State Examination (MMSE) scores, a Random Forest Regressor achieved a mean absolute MMSE error of 3.7 (3.7-3.8), with comparable performance of 3.3 (3.1-3.5) on pilot data. Linguistic features associated with higher ADRD risk included increased use of pronouns and adverbs, greater disfluency, reduced analytical thinking, lower lexical diversity and fewer words reflecting a psychological state of completion. Our interpretable predictive modelling offers a novel approach for in-home integration with conversational AI to monitor cognitive health and triage higher-risk individuals, enabling earlier detection and intervention.

Title: Examining the Robustness of Large Language Models across Language Complexity

• Authors: Jiayi Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18738
• Pdf URL: https://arxiv.org/pdf/2501.18738
• Copy Paste: [[2501.18738]] Examining the Robustness of Large Language Models across Language Complexity(https://arxiv.org/abs/2501.18738)
• Keywords: robust, large language model
• Abstract: With the advancement of large language models (LLMs), an increasing number of student models have leveraged LLMs to analyze textual artifacts generated by students to understand and evaluate their learning. These student models typically employ pre-trained LLMs to vectorize text inputs into embeddings and then use the embeddings to train models to detect the presence or absence of a construct of interest. However, how reliable and robust are these models at processing language with different levels of complexity? In the context of learning where students may have different language backgrounds with various levels of writing skills, it is critical to examine the robustness of such models to ensure that these models work equally well for text with varying levels of language complexity. Coincidentally, a few (but limited) research studies show that the use of language can indeed impact the performance of LLMs. As such, in the current study, we examined the robustness of several LLM-based student models that detect student self-regulated learning (SRL) in math problem-solving. Specifically, we compared how the performance of these models vary using texts with high and low lexical, syntactic, and semantic complexity measured by three linguistic measures.

Title: Neural Graph Pattern Machine

• Authors: Zehong Wang, Zheyuan Zhang, Tianyi Ma, Nitesh V Chawla, Chuxu Zhang, Yanfang Ye
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.18739
• Pdf URL: https://arxiv.org/pdf/2501.18739
• Copy Paste: [[2501.18739]] Neural Graph Pattern Machine(https://arxiv.org/abs/2501.18739)
• Keywords: robust, interpretability
• Abstract: Graph learning tasks require models to comprehend essential substructure patterns relevant to downstream tasks, such as triadic closures in social networks and benzene rings in molecular graphs. Due to the non-Euclidean nature of graphs, existing graph neural networks (GNNs) rely on message passing to iteratively aggregate information from local neighborhoods. Despite their empirical success, message passing struggles to identify fundamental substructures, such as triangles, limiting its expressiveness. To overcome this limitation, we propose the Neural Graph Pattern Machine (GPM), a framework designed to learn directly from graph patterns. GPM efficiently extracts and encodes substructures while identifying the most relevant ones for downstream tasks. We also demonstrate that GPM offers superior expressivity and improved long-range information modeling compared to message passing. Empirical evaluations on node classification, link prediction, graph classification, and regression show the superiority of GPM over state-of-the-art baselines. Further analysis reveals its desirable out-of-distribution robustness, scalability, and interpretability. We consider GPM to be a step toward going beyond message passing.

Title: REDACTOR: eFPGA Redaction for DNN Accelerator Security

• Authors: Yazan Baddour, Ava Hedayatipour, Amin Rezaei
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2501.18740
• Pdf URL: https://arxiv.org/pdf/2501.18740
• Copy Paste: [[2501.18740]] REDACTOR: eFPGA Redaction for DNN Accelerator Security(https://arxiv.org/abs/2501.18740)
• Keywords: security
• Abstract: With the ever-increasing integration of artificial intelligence into daily life and the growing importance of well-trained models, the security of hardware accelerators supporting Deep Neural Networks (DNNs) has become paramount. As a promising solution to prevent hardware intellectual property theft, eFPGA redaction has emerged. This technique selectively conceals critical components of the design, allowing authorized users to restore functionality post-fabrication by inserting the correct bitstream. In this paper, we explore the redaction of DNN accelerators using eFPGAs, from specification to physical design implementation. Specifically, we investigate the selection of critical DNN modules for redaction using both regular and fracturable look-up tables. We perform synthesis, timing verification, and place & route on redacted DNN accelerators. Furthermore, we evaluate the overhead of incorporating eFPGAs into DNN accelerators in terms of power, area, and delay, finding it reasonable given the security benefits.

Title: Synthetic Data Generation for Augmenting Small Samples

• Authors: Dan Liu, Samer El Kababji, Nicholas Mitsakakis, Lisa Pilgram, Thomas Walters, Mark Clemons, Greg Pond, Alaa El-Hussuna, Khaled El Emam
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18741
• Pdf URL: https://arxiv.org/pdf/2501.18741
• Copy Paste: [[2501.18741]] Synthetic Data Generation for Augmenting Small Samples(https://arxiv.org/abs/2501.18741)
• Keywords: generative
• Abstract: Small datasets are common in health research. However, the generalization performance of machine learning models is suboptimal when the training datasets are small. To address this, data augmentation is one solution. Augmentation increases sample size and is seen as a form of regularization that increases the diversity of small datasets, leading them to perform better on unseen data. We found that augmentation improves prognostic performance for datasets that: have fewer observations, with smaller baseline AUC, have higher cardinality categorical variables, and have more balanced outcome variables. No specific generative model consistently outperformed the others. We developed a decision support model that can be used to inform analysts if augmentation would be useful. For seven small application datasets, augmenting the existing data results in an increase in AUC between 4.31% (AUC from 0.71 to 0.75) and 43.23% (AUC from 0.51 to 0.73), with an average 15.55% relative improvement, demonstrating the nontrivial impact of augmentation on small datasets (p=0.0078). Augmentation AUC was higher than resampling only AUC (p=0.016). The diversity of augmented datasets was higher than the diversity of resampled datasets (p=0.046).

Title: Revisiting Projection-based Data Transfer for Cross-Lingual Named Entity Recognition in Low-Resource Languages

• Authors: Andrei Politov, Oleh Shkalikov, René Jäkel, Michael Färber
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.18750
• Pdf URL: https://arxiv.org/pdf/2501.18750
• Copy Paste: [[2501.18750]] Revisiting Projection-based Data Transfer for Cross-Lingual Named Entity Recognition in Low-Resource Languages(https://arxiv.org/abs/2501.18750)
• Keywords: robust
• Abstract: Cross-lingual Named Entity Recognition (NER) leverages knowledge transfer between languages to identify and classify named entities, making it particularly useful for low-resource languages. We show that the data-based cross-lingual transfer method is an effective technique for crosslingual NER and can outperform multilingual language models for low-resource languages. This paper introduces two key enhancements to the annotation projection step in cross-lingual NER for low-resource languages. First, we explore refining word alignments using back-translation to improve accuracy. Second, we present a novel formalized projection approach of matching source entities with extracted target candidates. Through extensive experiments on two datasets spanning 57 languages, we demonstrated that our approach surpasses existing projectionbased methods in low-resource settings. These findings highlight the robustness of projection-based data transfer as an alternative to model-based methods for crosslingual named entity recognition in lowresource languages.

Title: INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation

• Authors: Jian Hu, Zixu Cheng, Shaogang Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18753
• Pdf URL: https://arxiv.org/pdf/2501.18753
• Copy Paste: [[2501.18753]] INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation(https://arxiv.org/abs/2501.18753)
• Keywords: robust, segmentation
• Abstract: Task-generic promptable image segmentation aims to achieve segmentation of diverse samples under a single task description by utilizing only one task-generic prompt. Current methods leverage the generalization capabilities of Vision-Language Models (VLMs) to infer instance-specific prompts from these task-generic prompts in order to guide the segmentation process. However, when VLMs struggle to generalise to some image instances, predicting instance-specific prompts becomes poor. To solve this problem, we introduce \textbf{I}nstance-specific \textbf{N}egative Mining for \textbf{T}ask-Generic Promptable Segmentation (\textbf{INT}). The key idea of INT is to adaptively reduce the influence of irrelevant (negative) prior knowledge whilst to increase the use the most plausible prior knowledge, selected by negative mining with higher contrast, in order to optimise instance-specific prompts generation. Specifically, INT consists of two components: (1) instance-specific prompt generation, which progressively fliters out incorrect information in prompt generation; (2) semantic mask generation, which ensures each image instance segmentation matches correctly the semantics of the instance-specific prompts. INT is validated on six datasets, including camouflaged objects and medical images, demonstrating its effectiveness, robustness and scalability.

Title: Probabilistic Joint Recovery Method for CO$_2$ Plume Monitoring

• Authors: Zijun Deng, Rafael Orozco, Abhinav Prakash Gahlot, Felix J. Herrmann
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2501.18761
• Pdf URL: https://arxiv.org/pdf/2501.18761
• Copy Paste: [[2501.18761]] Probabilistic Joint Recovery Method for CO$_2$ Plume Monitoring(https://arxiv.org/abs/2501.18761)
• Keywords: generative
• Abstract: Reducing CO$_2$ emissions is crucial to mitigating climate change. Carbon Capture and Storage (CCS) is one of the few technologies capable of achieving net-negative CO$_2$ emissions. However, predicting fluid flow patterns in CCS remains challenging due to uncertainties in CO$_2$ plume dynamics and reservoir properties. Building on existing seismic imaging methods like the Joint Recovery Method (JRM), which lacks uncertainty quantification, we propose the Probabilistic Joint Recovery Method (pJRM). By estimating posterior distributions across surveys using a shared generative model, pJRM provides uncertainty information to improve risk assessment in CCS projects.

Title: Navigating the Fragrance space Via Graph Generative Models And Predicting Odors

• Authors: Mrityunjay Sharma, Sarabeshwar Balaji, Pinaki Saha, Ritesh Kumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18777
• Pdf URL: https://arxiv.org/pdf/2501.18777
• Copy Paste: [[2501.18777]] Navigating the Fragrance space Via Graph Generative Models And Predicting Odors(https://arxiv.org/abs/2501.18777)
• Keywords: interpretability, generative
• Abstract: We explore a suite of generative modelling techniques to efficiently navigate and explore the complex landscapes of odor and the broader chemical space. Unlike traditional approaches, we not only generate molecules but also predict the odor likeliness with ROC AUC score of 0.97 and assign probable odor labels. We correlate odor likeliness with physicochemical features of molecules using machine learning techniques and leverage SHAP (SHapley Additive exPlanations) to demonstrate the interpretability of the function. The whole process involves four key stages: molecule generation, stringent sanitization checks for molecular validity, fragrance likeliness screening and odor prediction of the generated molecules. By making our code and trained models publicly accessible, we aim to facilitate broader adoption of our research across applications in fragrance discovery and olfactory research.

Title: Gotta Hash 'Em All! Speeding Up Hash Functions for Zero-Knowledge Proof Applications

• Authors: Nojan Sheybani, Tengkai Gong, Anees Ahmed, Nges Brian Njungle, Michel Kinsy, Farinaz Koushanfar
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2501.18780
• Pdf URL: https://arxiv.org/pdf/2501.18780
• Copy Paste: [[2501.18780]] Gotta Hash 'Em All! Speeding Up Hash Functions for Zero-Knowledge Proof Applications(https://arxiv.org/abs/2501.18780)
• Keywords: security
• Abstract: Collision-resistant cryptographic hash functions (CRHs) are crucial for security in modern systems but are optimized for standard CPUs. While heavily used in zero-knowledge proof (ZKP) applications, traditional CRHs are inefficient in the ZK domain. ZK-friendly hashes have been developed but struggle on consumer hardware due to a lack of specialized ZK-specific hardware. To address this, we present HashEmAll, a novel collection of FPGA-based realizations of three ZK-friendly hash functions: Griffin, Rescue-Prime, and Reinforced Concrete. Each hash offers different optimization focuses, allowing users to choose based on the constraints of their applications. Through our ZK-optimized arithmetic functions on reconfigurable hardware, HashEmAll outperforms CPU implementations by up to $23\times$ with lower power consumption and compatibility with accessible FPGAs.

Title: RUN: Reversible Unfolding Network for Concealed Object Segmentation

• Authors: Chunming He, Rihan Zhang, Fengyang Xiao, Chenyu Fang, Longxiang Tang, Yulun Zhang, Linghe Kong, Deng-Ping Fan, Kai Li, Sina Farsiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18783
• Pdf URL: https://arxiv.org/pdf/2501.18783
• Copy Paste: [[2501.18783]] RUN: Reversible Unfolding Network for Concealed Object Segmentation(https://arxiv.org/abs/2501.18783)
• Keywords: extraction, segmentation
• Abstract: Existing concealed object segmentation (COS) methods frequently utilize reversible strategies to address uncertain regions. However, these approaches are typically restricted to the mask domain, leaving the potential of the RGB domain underexplored. To address this, we propose the Reversible Unfolding Network (RUN), which applies reversible strategies across both mask and RGB domains through a theoretically grounded framework, enabling accurate segmentation. RUN first formulates a novel COS model by incorporating an extra residual sparsity constraint to minimize segmentation uncertainties. The iterative optimization steps of the proposed model are then unfolded into a multistage network, with each step corresponding to a stage. Each stage of RUN consists of two reversible modules: the Segmentation-Oriented Foreground Separation (SOFS) module and the Reconstruction-Oriented Background Extraction (ROBE) module. SOFS applies the reversible strategy at the mask level and introduces Reversible State Space to capture non-local information. ROBE extends this to the RGB domain, employing a reconstruction network to address conflicting foreground and background regions identified as distortion-prone areas, which arise from their separate estimation by independent modules. As the stages progress, RUN gradually facilitates reversible modeling of foreground and background in both the mask and RGB domains, directing the network's attention to uncertain regions and mitigating false-positive and false-negative results. Extensive experiments demonstrate the superior performance of RUN and highlight the potential of unfolding-based frameworks for COS and other high-level vision tasks. We will release the code and models.

Title: Bayesian Optimization with Preference Exploration by Monotonic Neural Network Ensemble

• Authors: Hanyang Wang, Juergen Branke, Matthias Poloczek
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18792
• Pdf URL: https://arxiv.org/pdf/2501.18792
• Copy Paste: [[2501.18792]] Bayesian Optimization with Preference Exploration by Monotonic Neural Network Ensemble(https://arxiv.org/abs/2501.18792)
• Keywords: robust
• Abstract: Many real-world black-box optimization problems have multiple conflicting objectives. Rather than attempting to approximate the entire set of Pareto-optimal solutions, interactive preference learning allows to focus the search on the most relevant subset. However, few previous studies have exploited the fact that utility functions are usually monotonic. In this paper, we address the Bayesian Optimization with Preference Exploration (BOPE) problem and propose using a neural network ensemble as a utility surrogate model. This approach naturally integrates monotonicity and supports pairwise comparison data. Our experiments demonstrate that the proposed method outperforms state-of-the-art approaches and exhibits robustness to noise in utility evaluations. An ablation study highlights the critical role of monotonicity in enhancing performance.

Title: OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization

• Authors: Kelvin Kan, Xingjian Li, Stanley Osher
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18793
• Pdf URL: https://arxiv.org/pdf/2501.18793
• Copy Paste: [[2501.18793]] OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization(https://arxiv.org/abs/2501.18793)
• Keywords: transformer
• Abstract: Transformers have achieved state-of-the-art performance in numerous tasks. In this paper, we propose a continuous-time formulation of transformers. Specifically, we consider a dynamical system whose governing equation is parametrized by transformer blocks. We leverage optimal transport theory to regularize the training problem, which enhances stability in training and improves generalization of the resulting model. Moreover, we demonstrate in theory that this regularization is necessary as it promotes uniqueness and regularity of solutions. Our model is flexible in that almost any existing transformer architectures can be adopted to construct the dynamical system with only slight modifications to the existing code. We perform extensive numerical experiments on tasks motivated by natural language processing, image classification, and point cloud classification. Our experimental results show that the proposed method improves the performance of its discrete counterpart and outperforms relevant comparing models.

Title: Rope to Nope and Back Again: A New Hybrid Attention Strategy

• Authors: Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, Acyr Locatelli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18795
• Pdf URL: https://arxiv.org/pdf/2501.18795
• Copy Paste: [[2501.18795]] Rope to Nope and Back Again: A New Hybrid Attention Strategy(https://arxiv.org/abs/2501.18795)
• Keywords: transformer, large language model
• Abstract: Long-context large language models (LLMs) have achieved remarkable advancements, driven by techniques like Rotary Position Embedding (RoPE) (Su et al., 2023) and its extensions (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023). By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPE-based methods exhibit performance limitations when applied to extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm), identifying their strengths and shortcomings in long-context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long-context performance, providing valuable insights for architectural design. Building on these findings, we propose a novel architectural based on a hybrid attention mechanism that not only surpasses conventional RoPE-based transformer models in long context tasks but also achieves competitive performance on benchmarks requiring shorter context lengths.

Title: Compositional Generalization Requires More Than Disentangled Representations

• Authors: Qiyao Liang, Daoyuan Qian, Liu Ziyin, Ila Fiete
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18797
• Pdf URL: https://arxiv.org/pdf/2501.18797
• Copy Paste: [[2501.18797]] Compositional Generalization Requires More Than Disentangled Representations(https://arxiv.org/abs/2501.18797)
• Keywords: robust, generative
• Abstract: Composition-the ability to generate myriad variations from finite means-is believed to underlie powerful generalization. However, compositional generalization remains a key challenge for deep learning. A widely held assumption is that learning disentangled (factorized) representations naturally supports this kind of extrapolation. Yet, empirical results are mixed, with many generative models failing to recognize and compose factors to generate out-of-distribution (OOD) samples. In this work, we investigate a controlled 2D Gaussian "bump" generation task, demonstrating that standard generative architectures fail in OOD regions when training with partial data, even when supplied with fully disentangled $(x, y)$ coordinates, re-entangling them through subsequent layers. By examining the model's learned kernels and manifold geometry, we show that this failure reflects a "memorization" strategy for generation through the superposition of training data rather than by combining the true factorized features. We show that models forced-through architectural modifications with regularization or curated training data-to create disentangled representations in the full-dimensional representational (pixel) space can be highly data-efficient and effective at learning to compose in OOD regions. These findings underscore that bottlenecks with factorized/disentangled representations in an abstract representation are insufficient: the model must actively maintain or induce factorization directly in the representational space in order to achieve robust compositional generalization.

Title: Every Image Listens, Every Image Dances: Music-Driven Image Animation

• Authors: Zhikang Dong, Weituo Hao, Ju-Chiang Wang, Peng Zhang, Pawel Polak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18801
• Pdf URL: https://arxiv.org/pdf/2501.18801
• Copy Paste: [[2501.18801]] Every Image Listens, Every Image Dances: Music-Driven Image Animation(https://arxiv.org/abs/2501.18801)
• Keywords: robust, diffusion
• Abstract: Image animation has become a promising area in multimodal research, with a focus on generating videos from reference images. While prior work has largely emphasized generic video generation guided by text, music-driven dance video generation remains underexplored. In this paper, we introduce MuseDance, an innovative end-to-end model that animates reference images using both music and text inputs. This dual input enables MuseDance to generate personalized videos that follow text descriptions and synchronize character movements with the music. Unlike existing approaches, MuseDance eliminates the need for complex motion guidance inputs, such as pose or depth sequences, making flexible and creative video generation accessible to users of all expertise levels. To advance research in this field, we present a new multimodal dataset comprising 2,904 dance videos with corresponding background music and text descriptions. Our approach leverages diffusion-based methods to achieve robust generalization, precise control, and temporal consistency, setting a new baseline for the music-driven image animation task.

Title: Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion

• Authors: Vitor Guizilini, Muhammad Zubair Irshad, Dian Chen, Greg Shakhnarovich, Rares Ambrus
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18804
• Pdf URL: https://arxiv.org/pdf/2501.18804
• Copy Paste: [[2501.18804]] Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion(https://arxiv.org/abs/2501.18804)
• Keywords: diffusion
• Abstract: Current methods for 3D scene reconstruction from sparse posed images employ intermediate 3D representations such as neural fields, voxel grids, or 3D Gaussians, to achieve multi-view consistent scene appearance and geometry. In this paper we introduce MVGD, a diffusion-based architecture capable of direct pixel-level generation of images and depth maps from novel viewpoints, given an arbitrary number of input views. Our method uses raymap conditioning to both augment visual features with spatial information from different viewpoints, as well as to guide the generation of images and depth maps from novel views. A key aspect of our approach is the multi-task generation of images and depth maps, using learnable task embeddings to guide the diffusion process towards specific modalities. We train this model on a collection of more than 60 million multi-view samples from publicly available datasets, and propose techniques to enable efficient and consistent learning in such diverse conditions. We also propose a novel strategy that enables the efficient training of larger models by incrementally fine-tuning smaller ones, with promising scaling behavior. Through extensive experiments, we report state-of-the-art results in multiple novel view synthesis benchmarks, as well as multi-view stereo and video depth estimation.

Title: Learning Hamiltonian Dynamics with Bayesian Data Assimilation

• Authors: Taehyeun Kim, Tae-Geun Kim, Anouck Girard, Ilya Kolmanovsky
• Subjects: cs.LG, cs.RO, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.18808
• Pdf URL: https://arxiv.org/pdf/2501.18808
• Copy Paste: [[2501.18808]] Learning Hamiltonian Dynamics with Bayesian Data Assimilation(https://arxiv.org/abs/2501.18808)
• Keywords: robust
• Abstract: In this paper, we develop a neural network-based approach for time-series prediction in unknown Hamiltonian dynamical systems. Our approach leverages a surrogate model and learns the system dynamics using generalized coordinates (positions) and their conjugate momenta while preserving a constant Hamiltonian. To further enhance long-term prediction accuracy, we introduce an Autoregressive Hamiltonian Neural Network, which incorporates autoregressive prediction errors into the training objective. Additionally, we employ Bayesian data assimilation to refine predictions in real-time using online measurement data. Numerical experiments on a spring-mass system and highly elliptic orbits under gravitational perturbations demonstrate the effectiveness of the proposed method, highlighting its potential for accurate and robust long-term predictions.

Title: Quest Love: Do Blockchain Points Build Loyalty or Just Feed the Bots?

• Authors: Joseph Al-Chami, Jeremy Clark
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18810
• Pdf URL: https://arxiv.org/pdf/2501.18810
• Copy Paste: [[2501.18810]] Quest Love: Do Blockchain Points Build Loyalty or Just Feed the Bots?(https://arxiv.org/abs/2501.18810)
• Keywords: protect
• Abstract: Blockchain ecosystems -- such as those built around chains, layers, and services -- try to engage users for a variety of reasons: user education, growing and protecting their market share, climbing metric-measuring leaderboards with competing systems, demonstrating usage to investors, and identifying worthy recipients for newly created tokens (airdrops). A popular approach is offering user quests: small tasks that can be completed by a user, exposing them to a common task they might want to do in the future, and rewarding them for completion. In this paper, we capture blockchain data from one deployed quest system that offered 43 unique quests over 10 months with 80M completions. We use this data to offer insight about the factors that impact task completion: amount of reward, monetary value of reward, difficulty, and cost. We also discuss the role of farming and bots, and the factors that complicate distinguishing real users from automated scripts.

Title: Large Language Models as Common-Sense Heuristics

• Authors: Andrey Borro, Patricia J Riddle, Michael W Barley, Michael J Witbrock
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18816
• Pdf URL: https://arxiv.org/pdf/2501.18816
• Copy Paste: [[2501.18816]] Large Language Models as Common-Sense Heuristics(https://arxiv.org/abs/2501.18816)
• Keywords: large language model
• Abstract: While systems designed for solving planning tasks vastly outperform Large Language Models (LLMs) in this domain, they usually discard the rich semantic information embedded within task descriptions. In contrast, LLMs possess parametrised knowledge across a wide range of topics, enabling them to leverage the natural language descriptions of planning tasks in their solutions. However, current research in this direction faces challenges in generating correct and executable plans. Furthermore, these approaches depend on the LLM to output solutions in an intermediate language, which must be translated into the representation language of the planning task. We introduce a novel planning method, which leverages the parametrised knowledge of LLMs by using their output as a heuristic for Hill-Climbing Search. This approach is further enhanced by prompting the LLM to generate a solution estimate to guide the search. Our method outperforms the task success rate of similar systems within a common household environment by 22 percentage points, with consistently executable plans. All actions are encoded in their original representation, demonstrating that strong results can be achieved without an intermediate language, thus eliminating the need for a translation step.

Title: An Optimal Cascade Feature-Level Spatiotemporal Fusion Strategy for Anomaly Detection in CAN Bus

• Authors: Mohammad Fatahi, Danial Sadrian Zadeh, Benyamin Ghojogh, Behzad Moshiri, Otman Basir
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18821
• Pdf URL: https://arxiv.org/pdf/2501.18821
• Copy Paste: [[2501.18821]] An Optimal Cascade Feature-Level Spatiotemporal Fusion Strategy for Anomaly Detection in CAN Bus(https://arxiv.org/abs/2501.18821)
• Keywords: security, robust
• Abstract: Autonomous vehicles represent a revolutionary advancement driven by the integration of artificial intelligence within intelligent transportation systems. However, they remain vulnerable due to the absence of robust security mechanisms in the Controller Area Network (CAN) bus. In order to mitigate the security issue, many machine learning models and strategies have been proposed, which primarily focus on a subset of dominant patterns of anomalies and lack rigorous evaluation in terms of reliability and robustness. Therefore, to address the limitations of previous works and mitigate the security vulnerability in CAN bus, the current study develops a model based on the intrinsic nature of the problem to cover all dominant patterns of anomalies. To achieve this, a cascade feature-level fusion strategy optimized by a two-parameter genetic algorithm is proposed to combine temporal and spatial information. Subsequently, the model is evaluated using a paired t-test to ensure reliability and robustness. Finally, a comprehensive comparative analysis conducted on two widely used datasets advocates that the proposed model outperforms other models and achieves superior accuracy and F1-score, demonstrating the best performance among all models presented to date.

Title: Transcoders Beat Sparse Autoencoders for Interpretability

• Authors: Gonçalo Paulo, Stepan Shabalin, Nora Belrose
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18823
• Pdf URL: https://arxiv.org/pdf/2501.18823
• Copy Paste: [[2501.18823]] Transcoders Beat Sparse Autoencoders for Interpretability(https://arxiv.org/abs/2501.18823)
• Keywords: interpretability
• Abstract: Sparse autoencoders (SAEs) extract human-interpretable features from deep neural networks by transforming their activations into a sparse, higher dimensional latent space, and then reconstructing the activations from these latents. Transcoders are similar to SAEs, but they are trained to reconstruct the output of a component of a deep network given its input. In this work, we compare the features found by transcoders and SAEs trained on the same model and data, finding that transcoder features are significantly more interpretable. We also propose _skip transcoders_, which add an affine skip connection to the transcoder architecture, and show that these achieve lower reconstruction loss with no effect on interpretability.

Title: Memory-Efficient Fine-Tuning of Transformers via Token Selection

• Authors: Antoine Simoulin, Namyong Park, Xiaoyi Liu, Grey Yang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18824
• Pdf URL: https://arxiv.org/pdf/2501.18824
• Copy Paste: [[2501.18824]] Memory-Efficient Fine-Tuning of Transformers via Token Selection(https://arxiv.org/abs/2501.18824)
• Keywords: transformer
• Abstract: Fine-tuning provides an effective means to specialize pre-trained models for various downstream tasks. However, fine-tuning often incurs high memory overhead, especially for large transformer-based models, such as LLMs. While existing methods may reduce certain parts of the memory required for fine-tuning, they still require caching all intermediate activations computed in the forward pass to update weights during the backward pass. In this work, we develop TokenTune, a method to reduce memory usage, specifically the memory to store intermediate activations, in the fine-tuning of transformer-based models. During the backward pass, TokenTune approximates the gradient computation by backpropagating through just a subset of input tokens. Thus, with TokenTune, only a subset of intermediate activations are cached during the forward pass. Also, TokenTune can be easily combined with existing methods like LoRA, further reducing the memory cost. We evaluate our approach on pre-trained transformer models with up to billions of parameters, considering the performance on multiple downstream tasks such as text classification and question answering in a few-shot learning setup. Overall, TokenTune achieves performance on par with full fine-tuning or representative memory-efficient fine-tuning methods, while greatly reducing the memory footprint, especially when combined with other methods with complementary memory reduction mechanisms. We hope that our approach will facilitate the fine-tuning of large transformers, in specializing them for specific domains or co-training them with other neural components from a larger system. Our code is available at this https URL.

Title: Structural Embedding Projection for Contextual Large Language Model Inference

• Authors: Vincent Enoasmo, Cedric Featherstonehaugh, Xavier Konstantinopoulos, Zacharias Huntington
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18826
• Pdf URL: https://arxiv.org/pdf/2501.18826
• Copy Paste: [[2501.18826]] Structural Embedding Projection for Contextual Large Language Model Inference(https://arxiv.org/abs/2501.18826)
• Keywords: large language model
• Abstract: Structured embedding transformations offer a promising approach for enhancing the efficiency and coherence of language model inference. The introduction of Structural Embedding Projection (SEP) provides a mechanism for refining token representations through projection matrices that integrate hierarchical and relational dependencies. The mathematical formulation of SEP enables embedding spaces to capture structured contextual relationships, thereby improving semantic fidelity without significantly increasing computational overhead. Experimental evaluations conducted on a range of linguistic datasets revealed that SEP contributed to reductions in perplexity and enhanced contextual coherence, demonstrating its potential to refine language model outputs. Computational efficiency assessments highlighted variations across different datasets, suggesting that the integration of structured embeddings introduced dataset-dependent trade-offs between inference speed and representational richness. The qualitative analysis of generated responses indicated that SEP enhanced narrative consistency and topic alignment, leading to improved fluency in multi-sentence text generation. The modifications to embedding layers required precise optimization to ensure stable training dynamics, as the introduction of structured transformations altered the traditional representation-learning process. The architectural adjustments necessary for SEP implementation influenced inference latency and memory consumption, requiring a balance between efficiency gains and additional processing demands. The impact of SEP on lexical diversity suggested that embedding modifications influenced the model's vocabulary usage, reflecting a more context-aware selection of generated tokens.

Title: Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

• Authors: Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Logan Graham, Logan Howard, Nimit Kalra, Taesung Lee, Kevin Lin, Peter Lofgren, Francesco Mosconi, Clare O'Hara, Catherine Olsson, Linda Petrini, Samir Rajani, Nikhil Saxena, Alex Silverstein, Tanya Singh, Theodore Sumers, Leonard Tang, Kevin K. Troy, Constantin Weisser, Ruiqi Zhong, Giulio Zhou, Jan Leike, Jared Kaplan, Ethan Perez
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18837
• Pdf URL: https://arxiv.org/pdf/2501.18837
• Copy Paste: [[2501.18837]] Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming(https://arxiv.org/abs/2501.18837)
• Keywords: defense, attack, robust, large language model
• Abstract: Large language models (LLMs) are vulnerable to universal jailbreaks-prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Constitutional Classifiers: safeguards trained on synthetic data, generated by prompting LLMs with natural language rules (i.e., a constitution) specifying permitted and restricted content. In over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak that could extract information from an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries. On automated evaluations, enhanced classifiers demonstrated robust defense against held-out domain-specific jailbreaks. These classifiers also maintain deployment viability, with an absolute 0.38% increase in production-traffic refusals and a 23.7% inference overhead. Our work demonstrates that defending against universal jailbreaks while maintaining practical deployment viability is tractable.

Title: Partially Rewriting a Transformer in Natural Language

• Authors: Gonçalo Paulo, Nora Belrose
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18838
• Pdf URL: https://arxiv.org/pdf/2501.18838
• Copy Paste: [[2501.18838]] Partially Rewriting a Transformer in Natural Language(https://arxiv.org/abs/2501.18838)
• Keywords: interpretability, transformer, large language model
• Abstract: The greatest ambition of mechanistic interpretability is to completely rewrite deep neural networks in a format that is more amenable to human understanding, while preserving their behavior and performance. In this paper, we attempt to partially rewrite a large language model using simple natural language explanations. We first approximate one of the feedforward networks in the LLM with a wider MLP with sparsely activating neurons - a transcoder - and use an automated interpretability pipeline to generate explanations for these neurons. We then replace the first layer of this sparse MLP with an LLM-based simulator, which predicts the activation of each neuron given its explanation and the surrounding context. Finally, we measure the degree to which these modifications distort the model's final output. With our pipeline, the model's increase in loss is statistically similar to entirely replacing the sparse MLP output with the zero vector. We employ the same protocol, this time using a sparse autoencoder, on the residual stream of the same layer and obtain similar results. These results suggest that more detailed explanations are needed to improve performance substantially above the zero ablation baseline.

Title: Trading Inference-Time Compute for Adversarial Robustness

• Authors: Wojciech Zaremba, Evgenia Nitishinskaya, Boaz Barak, Stephanie Lin, Sam Toyer, Yaodong Yu, Rachel Dias, Eric Wallace, Kai Xiao, Johannes Heidecke, Amelia Glaese
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18841
• Pdf URL: https://arxiv.org/pdf/2501.18841
• Copy Paste: [[2501.18841]] Trading Inference-Time Compute for Adversarial Robustness(https://arxiv.org/abs/2501.18841)
• Keywords: attack, robust, large language model
• Abstract: We conduct experiments on the impact of increasing inference-time compute in reasoning models (specifically OpenAI o1-preview and o1-mini) on their robustness to adversarial attacks. We find that across a variety of attacks, increased inference-time compute leads to improved robustness. In many cases (with important exceptions), the fraction of model samples where the attack succeeds tends to zero as the amount of test-time compute grows. We perform no adversarial training for the tasks we study, and we increase inference-time compute by simply allowing the models to spend more compute on reasoning, independently of the form of attack. Our results suggest that inference-time compute has the potential to improve adversarial robustness for Large Language Models. We also explore new attacks directed at reasoning models, as well as settings where inference-time compute does not improve reliability, and speculate on the reasons for these as well as ways to address them.

Title: Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities

• Authors: Yaping Chai, Haoran Xie, Joe S. Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18845
• Pdf URL: https://arxiv.org/pdf/2501.18845
• Copy Paste: [[2501.18845]] Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities(https://arxiv.org/abs/2501.18845)
• Keywords: large language model
• Abstract: The increasing size and complexity of pre-trained language models have demonstrated superior performance in many applications, but they usually require large training datasets to be adequately trained. Insufficient training sets could unexpectedly make the model overfit and fail to cope with complex tasks. Large language models (LLMs) trained on extensive corpora have prominent text generation capabilities, which improve the quality and quantity of data and play a crucial role in data augmentation. Specifically, distinctive prompt templates are given in personalised tasks to guide LLMs in generating the required content. Recent promising retrieval-based techniques further improve the expressive performance of LLMs in data augmentation by introducing external knowledge to enable them to produce more grounded-truth data. This survey provides an in-depth analysis of data augmentation in LLMs, classifying the techniques into Simple Augmentation, Prompt-based Augmentation, Retrieval-based Augmentation and Hybrid Augmentation. We summarise the post-processing approaches in data augmentation, which contributes significantly to refining the augmented data and enabling the model to filter out unfaithful content. Then, we provide the common tasks and evaluation metrics. Finally, we introduce existing challenges and future opportunities that could bring further improvement to data augmentation.

Title: Project-and-Fuse: Improving RGB-D Semantic Segmentation via Graph Convolution Networks

• Authors: Xiaoyan Jiang, Bohan Wang, Xinlong Wan, Zhi Zhou, Hamido Fujita
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18851
• Pdf URL: https://arxiv.org/pdf/2501.18851
• Copy Paste: [[2501.18851]] Project-and-Fuse: Improving RGB-D Semantic Segmentation via Graph Convolution Networks(https://arxiv.org/abs/2501.18851)
• Keywords: extraction, segmentation
• Abstract: Most existing RGB-D semantic segmentation methods focus on the feature level fusion, including complex cross-modality and cross-scale fusion modules. However, these methods may cause misalignment problem in the feature fusion process and counter-intuitive patches in the segmentation results. Inspired by the popular pixel-node-pixel pipeline, we propose to 1) fuse features from two modalities in a late fusion style, during which the geometric feature injection is guided by texture feature prior; 2) employ Graph Neural Networks (GNNs) on the fused feature to alleviate the emergence of irregular patches by inferring patch relationship. At the 3D feature extraction stage, we argue that traditional CNNs are not efficient enough for depth maps. So, we encode depth map into normal map, after which CNNs can easily extract object surface this http URL projection matrix generation stage, we find the existence of Biased-Assignment and Ambiguous-Locality issues in the original pipeline. Therefore, we propose to 1) adopt the Kullback-Leibler Loss to ensure no missing important pixel features, which can be viewed as hard pixel mining process; 2) connect regions that are close to each other in the Euclidean space as well as in the semantic space with larger edge weights so that location informations can been considered. Extensive experiments on two public datasets, NYU-DepthV2 and SUN RGB-D, have shown that our approach can consistently boost the performance of RGB-D semantic segmentation task.

Title: FlexiCrackNet: A Flexible Pipeline for Enhanced Crack Segmentation with General Features Transfered from SAM

• Authors: Xinlong Wan, Xiaoyan Jiang, Guangsheng Luo, Ferdous Sohel, Jenqneng Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18855
• Pdf URL: https://arxiv.org/pdf/2501.18855
• Copy Paste: [[2501.18855]] FlexiCrackNet: A Flexible Pipeline for Enhanced Crack Segmentation with General Features Transfered from SAM(https://arxiv.org/abs/2501.18855)
• Keywords: robust, segmentation
• Abstract: Automatic crack segmentation is a cornerstone technology for intelligent visual perception modules in road safety maintenance and structural integrity systems. Existing deep learning models and ``pre-training + fine-tuning'' paradigms often face challenges of limited adaptability in resource-constrained environments and inadequate scalability across diverse data domains. To overcome these limitations, we propose FlexiCrackNet, a novel pipeline that seamlessly integrates traditional deep learning paradigms with the strengths of large-scale pre-trained models. At its core, FlexiCrackNet employs an encoder-decoder architecture to extract task-specific features. The lightweight EdgeSAM's CNN-based encoder is exclusively used as a generic feature extractor, decoupled from the fixed input size requirements of EdgeSAM. To harmonize general and domain-specific features, we introduce the information-Interaction gated attention mechanism (IGAM), which adaptively fuses multi-level features to enhance segmentation performance while mitigating irrelevant noise. This design enables the efficient transfer of general knowledge to crack segmentation tasks while ensuring adaptability to diverse input resolutions and resource-constrained environments. Experiments show that FlexiCrackNet outperforms state-of-the-art methods, excels in zero-shot generalization, computational efficiency, and segmentation robustness under challenging scenarios such as blurry inputs, complex backgrounds, and visually ambiguous artifacts. These advancements underscore the potential of FlexiCrackNet for real-world applications in automated crack detection and comprehensive structural health monitoring systems.

Title: DAPPER: A Performance-Attack-Resilient Tracker for RowHammer Defense

• Authors: Jeonghyun Woo, Prashant J. Nair
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18857
• Pdf URL: https://arxiv.org/pdf/2501.18857
• Copy Paste: [[2501.18857]] DAPPER: A Performance-Attack-Resilient Tracker for RowHammer Defense(https://arxiv.org/abs/2501.18857)
• Keywords: secure, security, defense, attack
• Abstract: RowHammer vulnerabilities pose a significant threat to modern DRAM-based systems, where rapid activation of DRAM rows can induce bit-flips in neighboring rows. To mitigate this, state-of-the-art host-side RowHammer mitigations typically rely on shared counters or tracking structures. While these optimizations benefit benign applications, they are vulnerable to Performance Attacks (Perf-Attacks), where adversaries exploit shared structures to reduce DRAM bandwidth for co-running benign applications by increasing DRAM accesses for RowHammer counters or triggering repetitive refreshes required for the early reset of structures, significantly degrading performance. In this paper, we propose secure hashing mechanisms to thwart adversarial attempts to capture the mapping of shared structures. We propose DAPPER, a novel low-cost tracker resilient to Perf-Attacks even at ultra-low RowHammer thresholds. We first present a secure hashing template in the form of DAPPER-S. We then develop DAPPER-H, an enhanced version of DAPPER-S, incorporating double-hashing, novel reset strategies, and mitigative refresh techniques. Our security analysis demonstrates the effectiveness of DAPPER-H against both RowHammer and Perf-Attacks. Experiments with 57 workloads from SPEC2006, SPEC2017, TPC, Hadoop, MediaBench, and YCSB show that, even at an ultra-low RowHammer threshold of 500, DAPPER-H incurs only a 0.9% slowdown in the presence of Perf-Attacks while using only 96KB of SRAM per 32GB of DRAM memory.

Title: BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning

• Authors: Han Zhong, Yutong Yin, Shenao Zhang, Xiaojun Xu, Yuanxin Liu, Yifei Zuo, Zhihan Liu, Boyi Liu, Sirui Zheng, Hongyi Guo, Liwei Wang, Mingyi Hong, Zhaoran Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18858
• Pdf URL: https://arxiv.org/pdf/2501.18858
• Copy Paste: [[2501.18858]] BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning(https://arxiv.org/abs/2501.18858)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model incorporating latent thinking processes and evaluation signals. Within this framework, we introduce the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm, which works in two steps. First, it generates high-quality rationales by approximating the optimal thinking process through reinforcement learning, using a novel reward shaping mechanism. Second, it enhances the base LLM by maximizing the joint probability of rationale generation with respect to the model's parameters. Theoretically, we demonstrate BRiTE's convergence at a rate of $1/T$ with $T$ representing the number of iterations. Empirical evaluations on math and coding benchmarks demonstrate that our approach consistently improves performance across different base models without requiring human-annotated thinking processes. In addition, BRiTE demonstrates superior performance compared to existing algorithms that bootstrap thinking processes use alternative methods such as rejection sampling, and can even match or exceed the results achieved through supervised fine-tuning with human-annotated data.

Title: QPRAC: Towards Secure and Practical PRAC-based Rowhammer Mitigation using Priority Queues

• Authors: Jeonghyun Woo, Shaopeng (Chris)Lin, Prashant J. Nair, Aamer Jaleel, Gururaj Saileshwar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18861
• Pdf URL: https://arxiv.org/pdf/2501.18861
• Copy Paste: [[2501.18861]] QPRAC: Towards Secure and Practical PRAC-based Rowhammer Mitigation using Priority Queues(https://arxiv.org/abs/2501.18861)
• Keywords: secure, security, attack
• Abstract: JEDEC has introduced the Per Row Activation Counting (PRAC) framework for DDR5 and future DRAMs to enable precise counting of DRAM row activations. PRAC enables a holistic mitigation of Rowhammer attacks even at ultra-low Rowhammer thresholds. PRAC uses an Alert Back-Off (ABO) protocol to request the memory controller to issue Rowhammer mitigation requests. However, recent PRAC implementations are either insecure or impractical. For example, Panopticon, the inspiration for PRAC, is rendered insecure if implemented per JEDEC's PRAC specification. On the other hand, the recent UPRAC proposal is impractical since it needs oracular knowledge of the `top-N' activated DRAM rows that require mitigation. This paper provides the first secure, scalable, and practical RowHammer solution using the PRAC framework. The crux of our proposal is the design of a priority-based service queue (PSQ) for mitigations that prioritizes pending mitigations based on activation counts to avoid the security risks of prior solutions. This provides principled security using the reactive ABO protocol. Furthermore, we co-design our PSQ, with opportunistic mitigation on Refresh Management (RFM) operations and proactive mitigation during refresh (REF), to limit the performance impact of ABO-based mitigations. QPRAC provides secure and practical RowHammer mitigation that scales to Rowhammer thresholds as low as 71 while incurring a 0.8% slowdown for benign workloads, which further reduces to 0% with proactive mitigations.

Title: REG: Rectified Gradient Guidance for Conditional Diffusion Models

• Authors: Zhengqi Gao, Kaiwen Zha, Tianyuan Zhang, Zihui Xue, Duane S. Boning
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18865
• Pdf URL: https://arxiv.org/pdf/2501.18865
• Copy Paste: [[2501.18865]] REG: Rectified Gradient Guidance for Conditional Diffusion Models(https://arxiv.org/abs/2501.18865)
• Keywords: diffusion
• Abstract: Guidance techniques are simple yet effective for improving conditional generation in diffusion models. Albeit their empirical success, the practical implementation of guidance diverges significantly from its theoretical motivation. In this paper, we reconcile this discrepancy by replacing the scaled marginal distribution target, which we prove theoretically invalid, with a valid scaled joint distribution objective. Additionally, we show that the established guidance implementations are approximations to the intractable optimal solution under no future foresight constraint. Building on these theoretical insights, we propose rectified gradient guidance (REG), a versatile enhancement designed to boost the performance of existing guidance methods. Experiments on 1D and 2D demonstrate that REG provides a better approximation to the optimal solution than prior guidance techniques, validating the proposed theoretical framework. Extensive experiments on class-conditional ImageNet and text-to-image generation tasks show that incorporating REG consistently improves FID and Inception/CLIP scores across various settings compared to its absence.

Title: Continuous-Time Analysis of Federated Averaging

• Authors: Tom Overman, Diego Klabjan
• Subjects: cs.LG, cs.DC, math.OC
• Abstract URL: https://arxiv.org/abs/2501.18870
• Pdf URL: https://arxiv.org/pdf/2501.18870
• Copy Paste: [[2501.18870]] Continuous-Time Analysis of Federated Averaging(https://arxiv.org/abs/2501.18870)
• Keywords: federate
• Abstract: Federated averaging (FedAvg) is a popular algorithm for horizontal federated learning (FL), where samples are gathered across different clients and are not shared with each other or a central server. Extensive convergence analysis of FedAvg exists for the discrete iteration setting, guaranteeing convergence for a range of loss functions and varying levels of data heterogeneity. We extend this analysis to the continuous-time setting where the global weights evolve according to a multivariate stochastic differential equation (SDE), which is the first time FedAvg has been studied from the continuous-time perspective. We use techniques from stochastic processes to establish convergence guarantees under different loss functions, some of which are more general than existing work in the discrete setting. We also provide conditions for which FedAvg updates to the server weights can be approximated as normal random variables. Finally, we use the continuous-time formulation to reveal generalization properties of FedAvg.

Title: Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling

• Authors: Macheng Shen, Chen Cheng
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18871
• Pdf URL: https://arxiv.org/pdf/2501.18871
• Copy Paste: [[2501.18871]] Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling(https://arxiv.org/abs/2501.18871)
• Keywords: diffusion, generative
• Abstract: Inspired by the ubiquitous use of differential equations to model continuous dynamics across diverse scientific and engineering domains, we propose a novel and intuitive approach to continuous sequence modeling. Our method interprets time-series data as \textit{discrete samples from an underlying continuous dynamical system}, and models its time evolution using Neural Stochastic Differential Equation (Neural SDE), where both the flow (drift) and diffusion terms are parameterized by neural networks. We derive a principled maximum likelihood objective and a \textit{simulation-free} scheme for efficient training of our Neural SDE model. We demonstrate the versatility of our approach through experiments on sequence modeling tasks across both embodied and generative AI. Notably, to the best of our knowledge, this is the first work to show that SDE-based continuous-time modeling also excels in such complex scenarios, and we hope that our work opens up new avenues for research of SDE models in high-dimensional and temporally intricate domains.

Title: Best Policy Learning from Trajectory Preference Feedback

• Authors: Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18873
• Pdf URL: https://arxiv.org/pdf/2501.18873
• Copy Paste: [[2501.18873]] Best Policy Learning from Trajectory Preference Feedback(https://arxiv.org/abs/2501.18873)
• Keywords: robust, generative
• Abstract: We address the problem of best policy identification in preference-based reinforcement learning (PbRL), where learning occurs from noisy binary preferences over trajectory pairs rather than explicit numerical rewards. This approach is useful for post-training optimization of generative AI models during multi-turn user interactions, where preference feedback is more robust than handcrafted reward models. In this setting, learning is driven by both an offline preference dataset -- collected from a rater of unknown 'competence' -- and online data collected with pure exploration. Since offline datasets may exhibit out-of-distribution (OOD) biases, principled online data collection is necessary. To address this, we propose Posterior Sampling for Preference Learning ($\mathsf{PSPL}$), a novel algorithm inspired by Top-Two Thompson Sampling, that maintains independent posteriors over the true reward model and transition dynamics. We provide the first theoretical guarantees for PbRL in this setting, establishing an upper bound on the simple Bayesian regret of $\mathsf{PSPL}$. Since the exact algorithm can be computationally impractical, we also provide an approximate version that outperforms existing baselines.

Title: Enforcing MAVLink Safety & Security Properties Via Refined Multiparty Session Types

• Authors: Arthur Amorim, Max Taylor, Gary T. Leavens, Bill Harrison, Lance Joneckis, Trevor Kann
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2501.18874
• Pdf URL: https://arxiv.org/pdf/2501.18874
• Copy Paste: [[2501.18874]] Enforcing MAVLink Safety & Security Properties Via Refined Multiparty Session Types(https://arxiv.org/abs/2501.18874)
• Keywords: security, attack
• Abstract: A compromised system component can issue message sequences that are perfectly legal while also leading the system itself into unsafe states. Such attacks are challenging to characterize, because message interfaces in standard languages define the individual messages possible but cannot express designers' intentions for how they should be used. We present initial results from ongoing work applying refined multiparty session types as a mechanism for expressing and enforcing proper message usage to exclude legal, but unsafe, sequences. We illustrate our approach by using refined multiparty session types to mitigate safety and security issues in the MAVLink protocol commonly used in UAVs.

Title: Self-Supervised Learning Using Nonlinear Dependence

• Authors: M.Hadi Sepanj, Benyamin Ghojogh, Paul Fieguth
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18875
• Pdf URL: https://arxiv.org/pdf/2501.18875
• Copy Paste: [[2501.18875]] Self-Supervised Learning Using Nonlinear Dependence(https://arxiv.org/abs/2501.18875)
• Keywords: robust
• Abstract: Self-supervised learning has gained significant attention in contemporary applications, particularly due to the scarcity of labeled data. While existing SSL methodologies primarily address feature variance and linear correlations, they often neglect the intricate relations between samples and the nonlinear dependencies inherent in complex data. In this paper, we introduce Correlation-Dependence Self-Supervised Learning (CDSSL), a novel framework that unifies and extends existing SSL paradigms by integrating both linear correlations and nonlinear dependencies, encapsulating sample-wise and feature-wise interactions. Our approach incorporates the Hilbert-Schmidt Independence Criterion (HSIC) to robustly capture nonlinear dependencies within a Reproducing Kernel Hilbert Space, enriching representation learning. Experimental evaluations on diverse benchmarks demonstrate the efficacy of CDSSL in improving representation quality.

Title: Distorting Embedding Space for Safety: A Defense Mechanism for Adversarially Robust Diffusion Models

• Authors: Jaesin Ahn, Heechul Jung
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18877
• Pdf URL: https://arxiv.org/pdf/2501.18877
• Copy Paste: [[2501.18877]] Distorting Embedding Space for Safety: A Defense Mechanism for Adversarially Robust Diffusion Models(https://arxiv.org/abs/2501.18877)
• Keywords: defense, attack, robust, diffusion
• Abstract: Text-to-image diffusion models show remarkable generation performance following text prompts, but risk generating Not Safe For Work (NSFW) contents from unsafe prompts. Existing approaches, such as prompt filtering or concept unlearning, fail to defend against adversarial attacks while maintaining benign image quality. In this paper, we propose a novel approach called Distorting Embedding Space (DES), a text encoder-based defense mechanism that effectively tackles these issues through innovative embedding space control. DES transforms unsafe embeddings, extracted from a text encoder using unsafe prompts, toward carefully calculated safe embedding regions to prevent unsafe contents generation, while reproducing the original safe embeddings. DES also neutralizes the nudity embedding, extracted using prompt ``nudity", by aligning it with neutral embedding to enhance robustness against adversarial attacks. These methods ensure both robust defense and high-quality image generation. Additionally, DES can be adopted in a plug-and-play manner and requires zero inference overhead, facilitating its deployment. Extensive experiments on diverse attack types, including black-box and white-box scenarios, demonstrate DES's state-of-the-art performance in both defense capability and benign image generation quality. Our model is available at this https URL.

Title: Building Bridges, Not Walls -- Advancing Interpretability by Unifying Feature, Data, and Model Component Attribution

• Authors: Shichang Zhang, Tessa Han, Usha Bhalla, Hima Lakkaraju
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18887
• Pdf URL: https://arxiv.org/pdf/2501.18887
• Copy Paste: [[2501.18887]] Building Bridges, Not Walls -- Advancing Interpretability by Unifying Feature, Data, and Model Component Attribution(https://arxiv.org/abs/2501.18887)
• Keywords: interpretability
• Abstract: The increasing complexity of AI systems has made understanding their behavior a critical challenge. Numerous methods have been developed to attribute model behavior to three key aspects: input features, training data, and internal model components. However, these attribution methods are studied and applied rather independently, resulting in a fragmented landscape of approaches and terminology. This position paper argues that feature, data, and component attribution methods share fundamental similarities, and bridging them can benefit interpretability research. We conduct a detailed analysis of successful methods across three domains and present a unified view to demonstrate that these seemingly distinct methods employ similar approaches, such as perturbations, gradients, and linear approximations, differing primarily in their perspectives rather than core techniques. Our unified perspective enhances understanding of existing attribution methods, identifies shared concepts and challenges, makes this field more accessible to newcomers, and highlights new directions not only for attribution and interpretability but also for broader AI research, including model editing, steering, and regulation.

Title: CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings

• Authors: Mohammad Al Olaimat, Serdar Bozdag
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18891
• Pdf URL: https://arxiv.org/pdf/2501.18891
• Copy Paste: [[2501.18891]] CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings(https://arxiv.org/abs/2501.18891)
• Keywords: robust, transformer
• Abstract: Electronic health records (EHRs) provide a comprehensive source of longitudinal patient data, encompassing structured modalities such as laboratory results, imaging data, and vital signs, and unstructured clinical notes. These datasets, after necessary preprocessing to clean and format the data for analysis, often remain in their raw EHR form, representing numerical or categorical values without further transformation into task-agnostic embeddings. While such raw EHR data enables predictive modeling, its reliance on manual feature engineering or downstream task-specific optimization limits its utility for general-purpose applications. Deep learning (DL) techniques, such as recurrent neural networks (RNNs) and Transformers, have facilitated predictive tasks like disease progression and diagnosis prediction. However, these methods often struggle to fully exploit the temporal and multimodal dependencies inherent in EHR data due to their reliance on pre-processed but untransformed raw EHR inputs. In this study, we introduce CAAT-EHR, a novel architecture designed to bridge this gap by generating robust, task-agnostic longitudinal embeddings from raw EHR data. CAAT-EHR leverages self- and cross-attention mechanisms in its encoder to integrate temporal and contextual relationships across multiple modalities, transforming the data into enriched embeddings that capture complex dependencies. An autoregressive decoder complements the encoder by predicting future time points data during pre-training, ensuring that the resulting embeddings maintain temporal consistency and alignment. CAAT-EHR eliminates the need for manual feature engineering and enables seamless transferability across diverse downstream tasks. Extensive evaluations on benchmark datasets, demonstrate the superiority of CAAT-EHR-generated embeddings over pre-processed raw EHR data and other baseline approaches.

Title: Lightspeed Geometric Dataset Distance via Sliced Optimal Transport

• Authors: Khai Nguyen, Hai Nguyen, Tuan Pham, Nhat Ho
• Subjects: cs.LG, cs.AI, stat.CO, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.18901
• Pdf URL: https://arxiv.org/pdf/2501.18901
• Copy Paste: [[2501.18901]] Lightspeed Geometric Dataset Distance via Sliced Optimal Transport(https://arxiv.org/abs/2501.18901)
• Keywords: robust
• Abstract: We introduce sliced optimal transport dataset distance (s-OTDD), a model-agnostic, embedding-agnostic approach for dataset comparison that requires no training, is robust to variations in the number of classes, and can handle disjoint label sets. The core innovation is Moment Transform Projection (MTP), which maps a label, represented as a distribution over features, to a real number. Using MTP, we derive a data point projection that transforms datasets into one-dimensional distributions. The s-OTDD is defined as the expected Wasserstein distance between the projected distributions, with respect to random projection parameters. Leveraging the closed form solution of one-dimensional optimal transport, s-OTDD achieves (near-)linear computational complexity in the number of data points and feature dimensions and is independent of the number of classes. With its geometrically meaningful projection, s-OTDD strongly correlates with the optimal transport dataset distance while being more efficient than existing dataset discrepancy measures. Moreover, it correlates well with the performance gap in transfer learning and classification accuracy in data augmentation.

Title: Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior

• Authors: Tongda Xu, Xiyan Cai, Xinjie Zhang, Xingtong Ge, Dailan He, Ming Sun, Jingjing Liu, Ya-Qin Zhang, Jian Li, Yan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18913
• Pdf URL: https://arxiv.org/pdf/2501.18913
• Copy Paste: [[2501.18913]] Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior(https://arxiv.org/abs/2501.18913)
• Keywords: diffusion
• Abstract: Recent advancements in diffusion models have been leveraged to address inverse problems without additional training, and Diffusion Posterior Sampling (DPS) (Chung et al., 2022a) is among the most popular approaches. Previous analyses suggest that DPS accomplishes posterior sampling by approximating the conditional score. While in this paper, we demonstrate that the conditional score approximation employed by DPS is not as effective as previously assumed, but rather aligns more closely with the principle of maximizing a posterior (MAP). This assertion is substantiated through an examination of DPS on 512x512 ImageNet images, revealing that: 1) DPS's conditional score estimation significantly diverges from the score of a well-trained conditional diffusion model and is even inferior to the unconditional score; 2) The mean of DPS's conditional score estimation deviates significantly from zero, rendering it an invalid score estimation; 3) DPS generates high-quality samples with significantly lower diversity. In light of the above findings, we posit that DPS more closely resembles MAP than a conditional score estimator, and accordingly propose the following enhancements to DPS: 1) we explicitly maximize the posterior through multi-step gradient ascent and projection; 2) we utilize a light-weighted conditional score estimator trained with only 100 images and 8 GPU hours. Extensive experimental results indicate that these proposed improvements significantly enhance DPS's performance. The source code for these improvements is provided in this https URL.

Title: Scaling Laws for Differentially Private Language Models

• Authors: Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A. Choquette-Choo, Badih Ghazi, George Kaissis, Ravi Kumar, Ruibo Liu, Da Yu, Chiyuan Zhang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18914
• Pdf URL: https://arxiv.org/pdf/2501.18914
• Copy Paste: [[2501.18914]] Scaling Laws for Differentially Private Language Models(https://arxiv.org/abs/2501.18914)
• Keywords: privacy, protect, large language model
• Abstract: Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive. LLMs also rely on large, high-quality training datasets, like those sourced from (sometimes sensitive) user data. Training models on this sensitive user data requires careful privacy protections like differential privacy (DP). However, the dynamics of DP training are significantly different, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute-privacy-utility tradeoffs and the optimal training configurations in many settings.

Title: LLM Program Optimization via Retrieval Augmented Search

• Authors: Sagnik Anupam, Alexander Shypula, Osbert Bastani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18916
• Pdf URL: https://arxiv.org/pdf/2501.18916
• Copy Paste: [[2501.18916]] LLM Program Optimization via Retrieval Augmented Search(https://arxiv.org/abs/2501.18916)
• Keywords: interpretability, large language model
• Abstract: With the advent of large language models (LLMs), there has been a great deal of interest in applying them to solve difficult programming tasks. Recent work has demonstrated their potential at program optimization, a key challenge in programming languages research. We propose a blackbox adaptation method called Retrieval Augmented Search (RAS) that performs beam search over candidate optimizations; at each step, it retrieves in-context examples from a given training dataset of slow-fast program pairs to guide the LLM. Critically, we find that performing contextual retrieval based on an LLM-generated natural language description significantly outperforms retrieval based on the source code. In addition, we propose a method called AEGIS for improving interpretability by decomposing training examples into "atomic edits" that are significantly more incremental in nature. We show that RAS performs 1.8$\times$ better than prior state-of-the-art blackbox adaptation strategies, and that AEGIS performs 1.37$\times$ better while performing significantly smaller edits.

Title: KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search

• Authors: Haoran Luo, Haihong E, Yikai Guo, Qika Lin, Xiaobao Wu, Xinyu Mu, Wenhao Liu, Meina Song, Yifan Zhu, Luu Anh Tuan
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2501.18922
• Pdf URL: https://arxiv.org/pdf/2501.18922
• Copy Paste: [[2501.18922]] KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search(https://arxiv.org/abs/2501.18922)
• Keywords: large language model
• Abstract: Knowledge Base Question Answering (KBQA) aims to answer natural language questions with a large-scale structured knowledge base (KB). Despite advancements with large language models (LLMs), KBQA still faces challenges in weak KB awareness, imbalance between effectiveness and efficiency, and high reliance on annotated data. To address these challenges, we propose KBQA-o1, a novel agentic KBQA method with Monte Carlo Tree Search (MCTS). It introduces a ReAct-based agent process for stepwise logical form generation with KB environment exploration. Moreover, it employs MCTS, a heuristic search method driven by policy and reward models, to balance agentic exploration's performance and search space. With heuristic exploration, KBQA-o1 generates high-quality annotations for further improvement by incremental fine-tuning. Experimental results show that KBQA-o1 outperforms previous low-resource KBQA methods with limited annotated data, boosting Llama-3.1-8B model's GrailQA F1 performance to 78.5% compared to 48.5% of the previous sota method with GPT-3.5-turbo.

Title: Training-free Quantum-Inspired Image Edge Extraction Method

• Authors: Arti Jain, Pradeep Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18929
• Pdf URL: https://arxiv.org/pdf/2501.18929
• Copy Paste: [[2501.18929]] Training-free Quantum-Inspired Image Edge Extraction Method(https://arxiv.org/abs/2501.18929)
• Keywords: robust, extraction, diffusion
• Abstract: Edge detection is a cornerstone of image processing, yet existing methods often face critical limitations. Traditional deep learning edge detection methods require extensive training datasets and fine-tuning, while classical techniques often fail in complex or noisy scenarios, limiting their real-world applicability. To address these limitations, we propose a training-free, quantum-inspired edge detection model. Our approach integrates classical Sobel edge detection, the Schrödinger wave equation refinement, and a hybrid framework combining Canny and Laplacian operators. By eliminating the need for training, the model is lightweight and adaptable to diverse applications. The Schrödinger wave equation refines gradient-based edge maps through iterative diffusion, significantly enhancing edge precision. The hybrid framework further strengthens the model by synergistically combining local and global features, ensuring robustness even under challenging conditions. Extensive evaluations on datasets like BIPED, Multicue, and NYUD demonstrate superior performance of the proposed model, achieving state-of-the-art metrics, including ODS, OIS, AP, and F-measure. Noise robustness experiments highlight its reliability, showcasing its practicality for real-world scenarios. Due to its versatile and adaptable nature, our model is well-suited for applications such as medical imaging, autonomous systems, and environmental monitoring, setting a new benchmark for edge detection.

Title: Deep Learning Model Inversion Attacks and Defenses: A Comprehensive Survey

• Authors: Wencheng Yang, Song Wang, Di Wu, Taotao Cai, Yanming Zhu, Shicheng Wei, Yiying Zhang, Xu Yang, Yan Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18934
• Pdf URL: https://arxiv.org/pdf/2501.18934
• Copy Paste: [[2501.18934]] Deep Learning Model Inversion Attacks and Defenses: A Comprehensive Survey(https://arxiv.org/abs/2501.18934)
• Keywords: security, privacy, defense, attack, biometric
• Abstract: The rapid adoption of deep learning in sensitive domains has brought tremendous benefits. However, this widespread adoption has also given rise to serious vulnerabilities, particularly model inversion (MI) attacks, posing a significant threat to the privacy and integrity of personal data. The increasing prevalence of these attacks in applications such as biometrics, healthcare, and finance has created an urgent need to understand their mechanisms, impacts, and defense methods. This survey aims to fill the gap in the literature by providing a structured and in-depth review of MI attacks and defense strategies. Our contributions include a systematic taxonomy of MI attacks, extensive research on attack techniques and defense mechanisms, and a discussion about the challenges and future research directions in this evolving field. By exploring the technical and ethical implications of MI attacks, this survey aims to offer insights into the impact of AI-powered systems on privacy, security, and trust. In conjunction with this survey, we have developed a comprehensive repository to support research on MI attacks and defenses. The repository includes state-of-the-art research papers, datasets, evaluation metrics, and other resources to meet the needs of both novice and experienced researchers interested in MI attacks and defenses, as well as the broader field of AI security and privacy. The repository will be continuously maintained to ensure its relevance and utility. It is accessible at this https URL.

Title: TabFSBench: Tabular Benchmark for Feature Shifts in Open Environment

• Authors: Zi-Jian Cheng, Zi-Yi Jia, Zhi Zhou, Lan-Zhe Guo, Yu-Feng Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18935
• Pdf URL: https://arxiv.org/pdf/2501.18935
• Copy Paste: [[2501.18935]] TabFSBench: Tabular Benchmark for Feature Shifts in Open Environment(https://arxiv.org/abs/2501.18935)
• Keywords: large language model
• Abstract: Tabular data is widely utilized in various machine learning tasks. Current tabular learning research predominantly focuses on closed environments, while in real-world applications, open environments are often encountered, where distribution and feature shifts occur, leading to significant degradation in model performance. Previous research has primarily concentrated on mitigating distribution shifts, whereas feature shifts, a distinctive and unexplored challenge of tabular data, have garnered limited attention. To this end, this paper conducts the first comprehensive study on feature shifts in tabular data and introduces the first tabular feature-shift benchmark (TabFSBench). TabFSBench evaluates impacts of four distinct feature-shift scenarios on four tabular model categories across various datasets and assesses the performance of large language models (LLMs) and tabular LLMs in the tabular benchmark for the first time. Our study demonstrates three main observations: (1) most tabular models have the limited applicability in feature-shift scenarios; (2) the shifted feature set importance has a linear relationship with model performance degradation; (3) model performance in closed environments correlates with feature-shift performance. Future research direction is also explored for each observation. TabFSBench is released for public access by using a few lines of Python codes at this https URL.

Title: Adaptive Prompt: Unlocking the Power of Visual Prompt Tuning

• Authors: Minh Le, Anh Nguyen, Huy Nguyen, Chau Nguyen, Nhat Ho
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18936
• Pdf URL: https://arxiv.org/pdf/2501.18936
• Copy Paste: [[2501.18936]] Adaptive Prompt: Unlocking the Power of Visual Prompt Tuning(https://arxiv.org/abs/2501.18936)
• Keywords: transformer
• Abstract: Visual Prompt Tuning (VPT) has recently emerged as a powerful method for adapting pre-trained vision models to downstream tasks. By introducing learnable prompt tokens as task-specific instructions, VPT effectively guides pre-trained transformer models with minimal overhead. Despite its empirical success, a comprehensive theoretical understanding of VPT remains an active area of research. Building on recent insights into the connection between mixture of experts and prompt-based approaches, we identify a key limitation in VPT: the restricted functional expressiveness in prompt formulation. To address this limitation, we propose Visual Adaptive Prompt Tuning (VAPT), a new generation of prompts that redefines prompts as adaptive functions of the input. Our theoretical analysis shows that this simple yet intuitive approach achieves optimal sample efficiency. Empirical results on VTAB-1K and FGVC further demonstrate VAPT's effectiveness, with performance gains of 7.34% and 1.04% over fully fine-tuning baselines, respectively. Notably, VAPT also surpasses VPT by a substantial margin while using fewer parameters. These results highlight both the effectiveness and efficiency of our method and pave the way for future research to explore the potential of adaptive prompts.

Title: TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction

• Authors: Sai Wang, Fan Ma, Xinyi Li, Hehe Fan, Yu Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18940
• Pdf URL: https://arxiv.org/pdf/2501.18940
• Copy Paste: [[2501.18940]] TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction(https://arxiv.org/abs/2501.18940)
• Keywords: interpretability
• Abstract: Recent advancements in LLMs have accelerated the development of dialogue generation across text and images, yet video-based dialogue generation remains underexplored and presents unique challenges. In this paper, we introduce Theme-aware Video Dialogue Crafting (TVDC), a novel task aimed at generating new dialogues that align with video content and adhere to user-specified themes. We propose TV-Dialogue, a novel multi-modal agent framework that ensures both theme alignment (i.e., the dialogue revolves around the theme) and visual consistency (i.e., the dialogue matches the emotions and behaviors of characters in the video) by enabling real-time immersive interactions among video characters, thereby accurately understanding the video content and generating new dialogue that aligns with the given themes. To assess the generated dialogues, we present a multi-granularity evaluation benchmark with high accuracy, interpretability and reliability, demonstrating the effectiveness of TV-Dialogue on self-collected dataset over directly using existing LLMs. Extensive experiments reveal that TV-Dialogue can generate dialogues for videos of any length and any theme in a zero-shot manner without training. Our findings underscore the potential of TV-Dialogue for various applications, such as video re-creation, film dubbing and its use in downstream multimodal tasks.

Title: Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them

• Authors: Anh Bui, Trang Vu, Long Vuong, Trung Le, Paul Montague, Tamas Abraham, Junae Kim, Dinh Phung
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18950
• Pdf URL: https://arxiv.org/pdf/2501.18950
• Copy Paste: [[2501.18950]] Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them(https://arxiv.org/abs/2501.18950)
• Keywords: diffusion
• Abstract: Concept erasure has emerged as a promising technique for mitigating the risk of harmful content generation in diffusion models by selectively unlearning undesirable concepts. The common principle of previous works to remove a specific concept is to map it to a fixed generic concept, such as a neutral concept or just an empty text prompt. In this paper, we demonstrate that this fixed-target strategy is suboptimal, as it fails to account for the impact of erasing one concept on the others. To address this limitation, we model the concept space as a graph and empirically analyze the effects of erasing one concept on the remaining concepts. Our analysis uncovers intriguing geometric properties of the concept space, where the influence of erasing a concept is confined to a local region. Building on this insight, we propose the Adaptive Guided Erasure (AGE) method, which \emph{dynamically} selects optimal target concepts tailored to each undesirable concept, minimizing unintended side effects. Experimental results show that AGE significantly outperforms state-of-the-art erasure methods on preserving unrelated concepts while maintaining effective erasure performance. Our code is published at {this https URL}.

Title: LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models

• Authors: Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, Wei-Shi Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18954
• Pdf URL: https://arxiv.org/pdf/2501.18954
• Copy Paste: [[2501.18954]] LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models(https://arxiv.org/abs/2501.18954)
• Keywords: large language model
• Abstract: Recent open-vocabulary detectors achieve promising performance with abundant region-level annotated data. In this work, we show that an open-vocabulary detector co-training with a large language model by generating image-level detailed captions for each image can further improve performance. To achieve the goal, we first collect a dataset, GroundingCap-1M, wherein each image is accompanied by associated grounding labels and an image-level detailed caption. With this dataset, we finetune an open-vocabulary detector with training objectives including a standard grounding loss and a caption generation loss. We take advantage of a large language model to generate both region-level short captions for each region of interest and image-level long captions for the whole image. Under the supervision of the large language model, the resulting detector, LLMDet, outperforms the baseline by a clear margin, enjoying superior open-vocabulary ability. Further, we show that the improved LLMDet can in turn build a stronger large multi-modal model, achieving mutual benefits. The code, model, and dataset is available at this https URL.

Title: Intrinsic Tensor Field Propagation in Large Language Models: A Novel Approach to Contextual Information Flow

• Authors: Alfred Bexley, Lukas Radcliffe, Giles Weatherstone, Joseph Sakau
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18957
• Pdf URL: https://arxiv.org/pdf/2501.18957
• Copy Paste: [[2501.18957]] Intrinsic Tensor Field Propagation in Large Language Models: A Novel Approach to Contextual Information Flow(https://arxiv.org/abs/2501.18957)
• Keywords: transformer, large language model
• Abstract: Context propagation remains a central challenge in language model architectures, particularly in tasks requiring the retention of long-range dependencies. Conventional attention mechanisms, while effective in many applications, exhibit limitations in maintaining coherent contextual representations over extended sequences due to their reliance on discrete token interactions. A novel approach is introduced through the formulation of Intrinsic Tensor Field Propagation (ITFP), which models contextual relationships as continuous tensor fields distributed across token embeddings. The propagation dynamics are governed through differential equations that enable a structured flow of contextual information, augmenting the standard attention mechanism to enhance coherence and recall. A series of experiments conducted on an open-source transformer-based model demonstrate that ITFP provides measurable improvements in contextual retention, dependency resolution, and inference stability across various linguistic structures. Comparisons with baseline models reveal a reduction in syntactic inconsistencies and factual errors, while ablation studies indicate that the choice of propagation depth and integration strength significantly impacts model performance. Additional evaluations assessing domain generalization suggest that ITFP effectively adapts across different text genres, reinforcing its applicability beyond conventional language modeling tasks. Although computational trade-offs are introduced through the inclusion of tensor field computations, empirical findings suggest that the benefits in accuracy and coherence outweigh the increased processing demands.

Title: Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Boostrapping

• Authors: Pu Yang, Yunzhen Feng, Ziyuan Chen, Yuhang Wu, Zhuoyuan Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18962
• Pdf URL: https://arxiv.org/pdf/2501.18962
• Copy Paste: [[2501.18962]] Spend Wisely: Maximizing Post-Training Gains in Iterative Synthetic Data Boostrapping(https://arxiv.org/abs/2501.18962)
• Keywords: diffusion, large language model
• Abstract: Modern foundation models often undergo iterative ``bootstrapping'' in their post-training phase: a model generates synthetic data, an external verifier filters out low-quality samples, and the high-quality subset is used for further fine-tuning. Over multiple iterations, the model's performance improves--raising a crucial question: how should the total budget on generation and training be allocated across iterations to maximize final performance? In this work, we develop a theoretical framework to analyze budget allocation strategies. Specifically, we show that constant policies fail to converge with high probability, while increasing policies--particularly exponential growth policies--exhibit significant theoretical advantages. Experiments on image denoising with diffusion probabilistic models and math reasoning with large language models show that both exponential and polynomial growth policies consistently outperform constant policies, with exponential policies often providing more stable performance.

Title: BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics

• Authors: Yuxuan Liu, Jingmin Sun, Hayden Schaeffer
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2501.18972
• Pdf URL: https://arxiv.org/pdf/2501.18972
• Copy Paste: [[2501.18972]] BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics(https://arxiv.org/abs/2501.18972)
• Keywords: robust, transformer
• Abstract: We introduce BCAT, a PDE foundation model designed for autoregressive prediction of solutions to two dimensional fluid dynamics problems. Our approach uses a block causal transformer architecture to model next frame predictions, leveraging previous frames as contextual priors rather than relying solely on sub-frames or pixel-based inputs commonly used in image generation methods. This block causal framework more effectively captures the spatial dependencies inherent in nonlinear spatiotemporal dynamics and physical phenomena. In an ablation study, next frame prediction demonstrated a 2.9x accuracy improvement over next token prediction. BCAT is trained on a diverse range of fluid dynamics datasets, including incompressible and compressible Navier-Stokes equations across various geometries and parameter regimes, as well as the shallow-water equations. The model's performance was evaluated on 6 distinct downstream prediction tasks and tested on about 8K trajectories to measure robustness on a variety of fluid dynamics simulations. BCAT achieved an average relative error of 1.92% across all evaluation tasks, outperforming prior approaches on standard benchmarks.

Title: GPO-VAE: Modeling Explainable Gene Perturbation Responses utilizing GRN-Aligned Parameter Optimization

• Authors: Seungheun Baek, Soyon Park, Yan Ting Chok, Mogan Gim, Jaewoo Kang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18973
• Pdf URL: https://arxiv.org/pdf/2501.18973
• Copy Paste: [[2501.18973]] GPO-VAE: Modeling Explainable Gene Perturbation Responses utilizing GRN-Aligned Parameter Optimization(https://arxiv.org/abs/2501.18973)
• Keywords: explainability
• Abstract: Motivation: Predicting cellular responses to genetic perturbations is essential for understanding biological systems and developing targeted therapeutic strategies. While variational autoencoders (VAEs) have shown promise in modeling perturbation responses, their limited explainability poses a significant challenge, as the learned features often lack clear biological meaning. Nevertheless, model explainability is one of the most important aspects in the realm of biological AI. One of the most effective ways to achieve explainability is incorporating the concept of gene regulatory networks (GRNs) in designing deep learning models such as VAEs. GRNs elicit the underlying causal relationships between genes and are capable of explaining the transcriptional responses caused by genetic perturbation treatments. Results: We propose GPO-VAE, an explainable VAE enhanced by GRN-aligned Parameter Optimization that explicitly models gene regulatory networks in the latent space. Our key approach is to optimize the learnable parameters related to latent perturbation effects towards GRN-aligned explainability. Experimental results on perturbation prediction show our model achieves state-of-the-art performance in predicting transcriptional responses across multiple benchmark datasets. Furthermore, additional results on evaluating the GRN inference task reveal our model's ability to generate meaningful GRNs compared to other methods. According to qualitative analysis, GPO-VAE posseses the ability to construct biologically explainable GRNs that align with experimentally validated regulatory pathways. GPO-VAE is available at this https URL

Title: Symmetric Pruning of Large Language Models

• Authors: Kai Yi, Peter Richtárik
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18980
• Pdf URL: https://arxiv.org/pdf/2501.18980
• Copy Paste: [[2501.18980]] Symmetric Pruning of Large Language Models(https://arxiv.org/abs/2501.18980)
• Keywords: large language model
• Abstract: Popular post-training pruning methods such as Wanda and RIA are known for their simple, yet effective, designs that have shown exceptional empirical performance. Wanda optimizes performance through calibrated activations during pruning, while RIA emphasizes the relative, rather than absolute, importance of weight elements. Despite their practical success, a thorough theoretical foundation explaining these outcomes has been lacking. This paper introduces new theoretical insights that redefine the standard minimization objective for pruning, offering a deeper understanding of the factors contributing to their success. Our study extends beyond these insights by proposing complementary strategies that consider both input activations and weight significance. We validate these approaches through rigorous experiments, demonstrating substantial enhancements over existing methods. Furthermore, we introduce a novel training-free fine-tuning approach $R^2$-DSnoT that incorporates relative weight importance and a regularized decision boundary within a dynamic pruning-and-growing framework, significantly outperforming strong baselines and establishing a new state of the art.

Title: OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation

• Authors: Yuchen Lin, Chenguo Lin, Jianjin Xu, Yadong Mu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18982
• Pdf URL: https://arxiv.org/pdf/2501.18982
• Copy Paste: [[2501.18982]] OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation(https://arxiv.org/abs/2501.18982)
• Keywords: diffusion
• Abstract: Recently, significant advancements have been made in the reconstruction and generation of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically assume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heterogeneous objects in real scenarios and tend to render less physically plausible animation given a wider range of objects. We propose OmniPhysGS for synthesizing a physics-based 3D dynamic scene composed of more general objects. A key design of OmniPhysGS is treating each 3D asset as a collection of constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensemble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementation, we define a scene by user-specified prompts and supervise the estimation of material weighting factors via a pretrained video diffusion model. Comprehensive experiments demonstrate that OmniPhysGS achieves more general and realistic physical dynamics across a broader spectrum of materials, including elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.

Title: Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images

• Authors: Zhengrui Guo, Qichen Sun, Jiabo Ma, Lishuang Feng, Jinzhuo Wang, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18984
• Pdf URL: https://arxiv.org/pdf/2501.18984
• Copy Paste: [[2501.18984]] Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images(https://arxiv.org/abs/2501.18984)
• Keywords: transformer
• Abstract: Whole slide image (WSI) analysis presents significant computational challenges due to the massive number of patches in gigapixel images. While transformer architectures excel at modeling long-range correlations through self-attention, their quadratic computational complexity makes them impractical for computational pathology applications. Existing solutions like local-global or linear self-attention reduce computational costs but compromise the strong modeling capabilities of full self-attention. In this work, we propose Querent, i.e., the query-aware long contextual dynamic modeling framework, which maintains the expressive power of full self-attention while achieving practical efficiency. Our method adaptively predicts which surrounding regions are most relevant for each patch, enabling focused yet unrestricted attention computation only with potentially important contexts. By using efficient region-wise metadata computation and importance estimation, our approach dramatically reduces computational overhead while preserving global perception to model fine-grained patch correlations. Through comprehensive experiments on biomarker prediction, gene mutation prediction, cancer subtyping, and survival analysis across over 10 WSI datasets, our method demonstrates superior performance compared to the state-of-the-art approaches. Code will be made available at this https URL.

Title: Visual Autoregressive Modeling for Image Super-Resolution

• Authors: Yunpeng Qu, Kun Yuan, Jinhua Hao, Kai Zhao, Qizhi Xie, Ming Sun, Chao Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18993
• Pdf URL: https://arxiv.org/pdf/2501.18993
• Copy Paste: [[2501.18993]] Visual Autoregressive Modeling for Image Super-Resolution(https://arxiv.org/abs/2501.18993)
• Keywords: robust, diffusion, generative
• Abstract: Image Super-Resolution (ISR) has seen significant progress with the introduction of remarkable generative models. However, challenges such as the trade-off issues between fidelity and realism, as well as computational complexity, have also posed limitations on their application. Building upon the tremendous success of autoregressive models in the language domain, we propose \textbf{VARSR}, a novel visual autoregressive modeling for ISR framework with the form of next-scale prediction. To effectively integrate and preserve semantic information in low-resolution images, we propose using prefix tokens to incorporate the condition. Scale-aligned Rotary Positional Encodings are introduced to capture spatial structures and the diffusion refiner is utilized for modeling quantization residual loss to achieve pixel-level fidelity. Image-based Classifier-free Guidance is proposed to guide the generation of more realistic images. Furthermore, we collect large-scale data and design a training process to obtain robust generative priors. Quantitative and qualitative results show that VARSR is capable of generating high-fidelity and high-realism images with more efficiency than diffusion-based methods. Our codes will be released at this https URL.

Title: Adversarial Attacks on AI-Generated Text Detection Models: A Token Probability-Based Approach Using Embeddings

• Authors: Ahmed K. Kadhim, Lei Jiao, Rishad Shafik, Ole-Christoffer Granmo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18998
• Pdf URL: https://arxiv.org/pdf/2501.18998
• Copy Paste: [[2501.18998]] Adversarial Attacks on AI-Generated Text Detection Models: A Token Probability-Based Approach Using Embeddings(https://arxiv.org/abs/2501.18998)
• Keywords: attack, robust
• Abstract: In recent years, text generation tools utilizing Artificial Intelligence (AI) have occasionally been misused across various domains, such as generating student reports or creative writings. This issue prompts plagiarism detection services to enhance their capabilities in identifying AI-generated content. Adversarial attacks are often used to test the robustness of AI-text generated detectors. This work proposes a novel textual adversarial attack on the detection models such as Fast-DetectGPT. The method employs embedding models for data perturbation, aiming at reconstructing the AI generated texts to reduce the likelihood of detection of the true origin of the texts. Specifically, we employ different embedding techniques, including the Tsetlin Machine (TM), an interpretable approach in machine learning for this purpose. By combining synonyms and embedding similarity vectors, we demonstrates the state-of-the-art reduction in detection scores against Fast-DetectGPT. Particularly, in the XSum dataset, the detection score decreased from 0.4431 to 0.2744 AUROC, and in the SQuAD dataset, it dropped from 0.5068 to 0.3532 AUROC.

Title: Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities

• Authors: Arjun Krishna, Erick Galinkin, Leon Derczynski, Jeffrey Martin
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.19012
• Pdf URL: https://arxiv.org/pdf/2501.19012
• Copy Paste: [[2501.19012]] Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities(https://arxiv.org/abs/2501.19012)
• Keywords: secure, attack, large language model
• Abstract: Large Language Models (LLMs) have become an essential tool in the programmer's toolkit, but their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain. In this work, we analyze package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies. By analyzing this package hallucination behaviour we find potential attacks and suggest defensive strategies to defend against these attacks. We discover that package hallucination rate is predicated not only on model choice, but also programming language, model size, and specificity of the coding task request. The Pareto optimality boundary between code generation performance and package hallucination is sparsely populated, suggesting that coding models are not being optimized for secure code. Additionally, we find an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages. Our metrics, findings and analyses provide a base for future models, securing AI-assisted software development workflows against package supply chain attacks.

Title: Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation

• Authors: Bin Zhu, Hui yan Qi, Yinxuan Gui, Jingjing Chen, Chong-Wah Ngo, Ee Peng Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19017
• Pdf URL: https://arxiv.org/pdf/2501.19017
• Copy Paste: [[2501.19017]] Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation(https://arxiv.org/abs/2501.19017)
• Keywords: robust, large language model
• Abstract: Multimodal Large Language Models (MLLMs) have exhibited remarkable advancements in integrating different modalities, excelling in complex understanding and generation tasks. Despite their success, MLLMs remain vulnerable to conversational adversarial inputs, particularly negation arguments. This paper systematically evaluates state-of-the-art MLLMs across diverse benchmarks, revealing significant performance drops when negation arguments are introduced to initially correct responses. We show critical vulnerabilities in the reasoning and alignment mechanisms of these models. Proprietary models such as GPT-4o and Claude-3.5-Sonnet demonstrate better resilience compared to open-source counterparts like Qwen2-VL and LLaVA. However, all evaluated MLLMs struggle to maintain logical consistency under negation arguments during conversation. This paper aims to offer valuable insights for improving the robustness of MLLMs against adversarial inputs, contributing to the development of more reliable and trustworthy multimodal AI systems.

Title: Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses

• Authors: Ahmed K. Kadhim, Lei Jiao, Rishad Shafik, Ole-Christoffer Granmo, Bimal Bhattarai
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19018
• Pdf URL: https://arxiv.org/pdf/2501.19018
• Copy Paste: [[2501.19018]] Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses(https://arxiv.org/abs/2501.19018)
• Keywords: interpretability
• Abstract: The Tsetlin Machine (TM) architecture has recently demonstrated effectiveness in Machine Learning (ML), particularly within Natural Language Processing (NLP). It has been utilized to construct word embedding using conjunctive propositional clauses, thereby significantly enhancing our understanding and interpretation of machine-derived decisions. The previous approach performed the word embedding over a sequence of input words to consolidate the information into a cohesive and unified representation. However, that approach encounters scalability challenges as the input size increases. In this study, we introduce a novel approach incorporating two-phase training to discover contextual embeddings of input sequences. Specifically, this method encapsulates the knowledge for each input word within the dataset's vocabulary, subsequently constructing embeddings for a sequence of input words utilizing the extracted knowledge. This technique not only facilitates the design of a scalable model but also preserves interpretability. Our experimental findings revealed that the proposed method yields competitive performance compared to the previous approaches, demonstrating promising results in contrast to human-generated benchmarks. Furthermore, we applied the proposed approach to sentiment analysis on the IMDB dataset, where the TM embedding and the TM classifier, along with other interpretable classifiers, offered a transparent end-to-end solution with competitive performance.

Title: On the Impact of Noise in Differentially Private Text Rewriting

• Authors: Stephen Meisenbacher, Maulik Chevli, Florian Matthes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19022
• Pdf URL: https://arxiv.org/pdf/2501.19022
• Copy Paste: [[2501.19022]] On the Impact of Noise in Differentially Private Text Rewriting(https://arxiv.org/abs/2501.19022)
• Keywords: privacy, protect
• Abstract: The field of text privatization often leverages the notion of $\textit{Differential Privacy}$ (DP) to provide formal guarantees in the rewriting or obfuscation of sensitive textual data. A common and nearly ubiquitous form of DP application necessitates the addition of calibrated noise to vector representations of text, either at the data- or model-level, which is governed by the privacy parameter $\varepsilon$. However, noise addition almost undoubtedly leads to considerable utility loss, thereby highlighting one major drawback of DP in NLP. In this work, we introduce a new sentence infilling privatization technique, and we use this method to explore the effect of noise in DP text rewriting. We empirically demonstrate that non-DP privatization techniques excel in utility preservation and can find an acceptable empirical privacy-utility trade-off, yet cannot outperform DP methods in empirical privacy protections. Our results highlight the significant impact of noise in current DP rewriting mechanisms, leading to a discussion of the merits and challenges of DP in NLP, as well as the opportunities that non-DP methods present.

Title: SynthmanticLiDAR: A Synthetic Dataset for Semantic Segmentation on LiDAR Imaging

• Authors: Javier Montalvo, Pablo Carballeira, Álvaro García-Martín
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19035
• Pdf URL: https://arxiv.org/pdf/2501.19035
• Copy Paste: [[2501.19035]] SynthmanticLiDAR: A Synthetic Dataset for Semantic Segmentation on LiDAR Imaging(https://arxiv.org/abs/2501.19035)
• Keywords: segmentation
• Abstract: Semantic segmentation on LiDAR imaging is increasingly gaining attention, as it can provide useful knowledge for perception systems and potential for autonomous driving. However, collecting and labeling real LiDAR data is an expensive and time-consuming task. While datasets such as SemanticKITTI have been manually collected and labeled, the introduction of simulation tools such as CARLA, has enabled the creation of synthetic datasets on demand. In this work, we present a modified CARLA simulator designed with LiDAR semantic segmentation in mind, with new classes, more consistent object labeling with their counterparts from real datasets such as SemanticKITTI, and the possibility to adjust the object class distribution. Using this tool, we have generated SynthmanticLiDAR, a synthetic dataset for semantic segmentation on LiDAR imaging, designed to be similar to SemanticKITTI, and we evaluate its contribution to the training process of different semantic segmentation algorithms by using a naive transfer learning approach. Our results show that incorporating SynthmanticLiDAR into the training process improves the overall performance of tested algorithms, proving the usefulness of our dataset, and therefore, our adapted CARLA simulator. The dataset and simulator are available in this https URL.

Title: Beyond Token Compression: A Training-Free Reduction Framework for Efficient Visual Processing in MLLMs

• Authors: Hongliang Li, Jiaxin Zhang, Wenhui Liao, Dezhi Peng, Kai Ding, Lianwen Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19036
• Pdf URL: https://arxiv.org/pdf/2501.19036
• Copy Paste: [[2501.19036]] Beyond Token Compression: A Training-Free Reduction Framework for Efficient Visual Processing in MLLMs(https://arxiv.org/abs/2501.19036)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) are typically based on decoder-only or cross-attention architectures. While decoder-only MLLMs outperform their cross-attention counterparts, they require significantly higher computational resources due to extensive self-attention and FFN operations on visual tokens. This raises the question: can we eliminate these expensive operations while maintaining the performance? To this end, we present a novel analysis framework to investigate the necessity of these costly operations in decoder-only MLLMs. Our framework introduces two key innovations: (1) Hollow Attention, which limits visual token interactions to local attention while maintaining visual-text associations, and (2) Probe-Activated Dynamic FFN, which selectively activates FFN parameters for visual tokens. Both methods do not require fine-tuning, which significantly enhances analysis efficiency. To assess the impact of applying these reductions across different proportions of layers, we developed a greedy search method that significantly narrows the search space. Experiments on state-of-the-art MLLMs reveal that applying our reductions to approximately half of the layers not only maintains but sometimes improves model performance, indicating significant computational redundancy in current architectures. Additionally, our method is orthogonal to existing token compression techniques, allowing for further combination to achieve greater computational reduction. Our findings may provide valuable insights for the design of more efficient future MLLMs. Our code will be publicly available at this https URL.

Title: Towards the Worst-case Robustness of Large Language Models

• Authors: Huanran Chen, Yinpeng Dong, Zeming Wei, Hang Su, Jun Zhu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19040
• Pdf URL: https://arxiv.org/pdf/2501.19040
• Copy Paste: [[2501.19040]] Towards the Worst-case Robustness of Large Language Models(https://arxiv.org/abs/2501.19040)
• Keywords: defense, attack, robust, large language model
• Abstract: Recent studies have revealed the vulnerability of Large Language Models (LLMs) to adversarial attacks, where the adversary crafts specific input sequences to induce harmful, violent, private, or incorrect outputs. Although various defenses have been proposed, they have not been evaluated by strong adaptive attacks, leaving the worst-case robustness of LLMs still intractable. By developing a stronger white-box attack, our evaluation results indicate that most typical defenses achieve nearly 0\% this http URL solve this, we propose \textit{DiffTextPure}, a general defense that diffuses the (adversarial) input prompt using any pre-defined smoothing distribution, and purifies the diffused input using a pre-trained language model. Theoretically, we derive tight robustness lower bounds for all smoothing distributions using Fractal Knapsack or 0-1 Knapsack solvers. Under this framework, we certify the robustness of a specific case -- smoothing LLMs using a uniform kernel -- against \textit{any possible attack} with an average $\ell_0$ perturbation of 2.02 or an average suffix length of 6.41.

Title: Self-Supervised Cross-Modal Text-Image Time Series Retrieval in Remote Sensing

• Authors: Genc Hoxha, Olivér Angyal, Begüm Demir
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19043
• Pdf URL: https://arxiv.org/pdf/2501.19043
• Copy Paste: [[2501.19043]] Self-Supervised Cross-Modal Text-Image Time Series Retrieval in Remote Sensing(https://arxiv.org/abs/2501.19043)
• Keywords: transformer
• Abstract: The development of image time series retrieval (ITSR) methods is a growing research interest in remote sensing (RS). Given a user-defined image time series (i.e., the query time series), the ITSR methods search and retrieve from large archives the image time series that have similar content to the query time series. The existing ITSR methods in RS are designed for unimodal retrieval problems, limiting their usability and versatility. To overcome this issue, as a first time in RS we introduce the task of cross-modal text-ITSR. In particular, we present a self-supervised cross-modal text-image time series retrieval (text-ITSR) method that enables the retrieval of image time series using text sentences as queries, and vice versa. In detail, we focus our attention on text-ITSR in pairs of images (i.e., bitemporal images). The proposed text-ITSR method consists of two key components: 1) modality-specific encoders to model the semantic content of bitemporal images and text sentences with discriminative features; and 2) modality-specific projection heads to align textual and image representations in a shared embedding space. To effectively model the temporal information within the bitemporal images, we introduce two fusion strategies: i) global feature fusion (GFF) strategy that combines global image features through simple yet effective operators; and ii) transformer-based feature fusion (TFF) strategy that leverages transformers for fine-grained temporal integration. Extensive experiments conducted on two benchmark RS archives demonstrate the effectiveness of the proposed method in accurately retrieving semantically relevant bitemporal images (or text sentences) to a query text sentence (or bitemporal image). The code of this work is publicly available at this https URL.

Title: Norm-Bounded Low-Rank Adaptation

• Authors: Ruigang Wang, Krishnamurthy Dvijotham, Ian R. Manchester
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19050
• Pdf URL: https://arxiv.org/pdf/2501.19050
• Copy Paste: [[2501.19050]] Norm-Bounded Low-Rank Adaptation(https://arxiv.org/abs/2501.19050)
• Keywords: privacy, robust
• Abstract: In this work, we propose norm-bounded low-rank adaptation (NB-LoRA) for parameter-efficient fine tuning. We introduce two parameterizations that allow explicit bounds on each singular value of the weight adaptation matrix, which can therefore satisfy any prescribed unitarily invariant norm bound, including the Schatten norms (e.g., nuclear, Frobenius, spectral norm). The proposed parameterizations are unconstrained and complete, i.e. they cover all matrices satisfying the prescribed rank and norm constraints. Experiments on vision fine-tuning benchmarks show that the proposed approach can achieve good adaptation performance while avoiding model catastrophic forgetting and also substantially improve robustness to a wide range of hyper-parameters, including adaptation rank, learning rate and number of training epochs. We also explore applications in privacy-preserving model merging and low-rank matrix completion.

Title: Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models

• Authors: Ruiyu Wang, Yu Yuan, Shizhao Sun, Jiang Bian
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19054
• Pdf URL: https://arxiv.org/pdf/2501.19054
• Copy Paste: [[2501.19054]] Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models(https://arxiv.org/abs/2501.19054)
• Keywords: large language model
• Abstract: Creating Computer-Aided Design (CAD) models requires significant expertise and effort. Text-to-CAD, which converts textual descriptions into CAD parametric sequences, is crucial in streamlining this process. Recent studies have utilized ground-truth parametric sequences, known as sequential signals, as supervision to achieve this goal. However, CAD models are inherently multimodal, comprising parametric sequences and corresponding rendered visual objects. Besides,the rendering process from parametric sequences to visual objects is many-to-one. Therefore, both sequential and visual signals are critical for effective training. In this work, we introduce CADFusion, a framework that uses Large Language Models (LLMs) as the backbone and alternates between two training stages: the sequential learning (SL) stage and the visual feedback (VF) stage. In the SL stage, we train LLMs using ground-truth parametric sequences, enabling the generation of logically coherent parametric sequences. In the VF stage, we reward parametric sequences that render into visually preferred objects and penalize those that do not, allowing LLMs to learn how rendered visual objects are perceived and evaluated. These two stages alternate throughout the training, ensuring balanced learning and preserving benefits of both signals. Experiments demonstrate that CADFusion significantly improves performance, both qualitatively and quantitatively.

Title: TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs

• Authors: Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, Dacheng Tao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19057
• Pdf URL: https://arxiv.org/pdf/2501.19057
• Copy Paste: [[2501.19057]] TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs(https://arxiv.org/abs/2501.19057)
• Keywords: large language model
• Abstract: Zeroth-order optimization (ZO) has demonstrated remarkable promise in efficient fine-tuning tasks for Large Language Models (LLMs). In particular, recent advances incorporate the low-rankness of gradients, introducing low-rank ZO estimators to further reduce GPU memory consumption. However, most existing works focus solely on the low-rankness of each individual gradient, overlooking a broader property shared by all gradients throughout the training, i.e., all gradients approximately reside within a similar subspace. In this paper, we consider two factors together and propose a novel low-rank ZO estimator, TeZO, which captures the low-rankness across both the model and temporal dimension. Specifically, we represent ZO perturbations along the temporal dimension as a 3D tensor and employ Canonical Polyadic Decomposition (CPD) to extract each low-rank 2D matrix, significantly reducing the training cost. TeZO can also be easily extended to the Adam variant while consuming less memory than MeZO-SGD, and requiring about only 35% memory of MeZO-Adam. Both comprehensive theoretical analysis and extensive experimental research have validated its efficiency, achieving SOTA-comparable results with lower overhead of time and memory.

Title: Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable Generations

• Authors: Dahye Kim, Deepti Ghadiyaram
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19066
• Pdf URL: https://arxiv.org/pdf/2501.19066
• Copy Paste: [[2501.19066]] Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable Generations(https://arxiv.org/abs/2501.19066)
• Keywords: attack, robust, diffusion, generative
• Abstract: Despite the remarkable progress in text-to-image generative models, they are prone to adversarial attacks and inadvertently generate unsafe, unethical content. Existing approaches often rely on fine-tuning models to remove specific concepts, which is computationally expensive, lack scalability, and/or compromise generation quality. In this work, we propose a novel framework leveraging k-sparse autoencoders (k-SAEs) to enable efficient and interpretable concept manipulation in diffusion models. Specifically, we first identify interpretable monosemantic concepts in the latent space of text embeddings and leverage them to precisely steer the generation away or towards a given concept (e.g., nudity) or to introduce a new concept (e.g., photographic style). Through extensive experiments, we demonstrate that our approach is very simple, requires no retraining of the base model nor LoRA adapters, does not compromise the generation quality, and is robust to adversarial prompt manipulations. Our method yields an improvement of $\mathbf{20.01\%}$ in unsafe concept removal, is effective in style manipulation, and is $\mathbf{\sim5}$x faster than current state-of-the-art.

Title: Improving vision-language alignment with graph spiking hybrid Networks

• Authors: Siyu Zhang, Heming Zheng, Yiming Wu, Yeming Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19069
• Pdf URL: https://arxiv.org/pdf/2501.19069
• Copy Paste: [[2501.19069]] Improving vision-language alignment with graph spiking hybrid Networks(https://arxiv.org/abs/2501.19069)
• Keywords: segmentation
• Abstract: To bridge the semantic gap between vision and language (VL), it is necessary to develop a good alignment strategy, which includes handling semantic diversity, abstract representation of visual information, and generalization ability of models. Recent works use detector-based bounding boxes or patches with regular partitions to represent visual semantics. While current paradigms have made strides, they are still insufficient for fully capturing the nuanced contextual relations among various objects. This paper proposes a comprehensive visual semantic representation module, necessitating the utilization of panoptic segmentation to generate coherent fine-grained semantic features. Furthermore, we propose a novel Graph Spiking Hybrid Network (GSHN) that integrates the complementary advantages of Spiking Neural Networks (SNNs) and Graph Attention Networks (GATs) to encode visual semantic information. Intriguingly, the model not only encodes the discrete and continuous latent variables of instances but also adeptly captures both local and global contextual features, thereby significantly enhancing the richness and diversity of semantic representations. Leveraging the spatiotemporal properties inherent in SNNs, we employ contrastive learning (CL) to enhance the similarity-based representation of embeddings. This strategy alleviates the computational overhead of the model and enriches meaningful visual representations by constructing positive and negative sample pairs. We design an innovative pre-training method, Spiked Text Learning (STL), which uses text features to improve the encoding ability of discrete semantics. Experiments show that the proposed GSHN exhibits promising results on multiple VL downstream tasks.

Title: Differentially Private Policy Gradient

• Authors: Alexandre Rio, Merwan Barlier, Igor Colin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19080
• Pdf URL: https://arxiv.org/pdf/2501.19080
• Copy Paste: [[2501.19080]] Differentially Private Policy Gradient(https://arxiv.org/abs/2501.19080)
• Keywords: privacy
• Abstract: Motivated by the increasing deployment of reinforcement learning in the real world, involving a large consumption of personal data, we introduce a differentially private (DP) policy gradient algorithm. We show that, in this setting, the introduction of Differential Privacy can be reduced to the computation of appropriate trust regions, thus avoiding the sacrifice of theoretical properties of the DP-less methods. Therefore, we show that it is possible to find the right trade-off between privacy noise and trust-region size to obtain a performant differentially private policy gradient algorithm. We then outline its performance empirically on various benchmarks. Our results and the complexity of the tasks addressed represent a significant improvement over existing DP algorithms in online RL.

Title: A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration

• Authors: Yuchen Hu, Xi Chen, Weidong Liu, Xiaojun Mao
• Subjects: cs.LG, cs.DC, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.19082
• Pdf URL: https://arxiv.org/pdf/2501.19082
• Copy Paste: [[2501.19082]] A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration(https://arxiv.org/abs/2501.19082)
• Keywords: diffusion
• Abstract: Distributed stochastic optimization algorithms can handle large-scale data simultaneously and accelerate model training. However, the sparsity of distributed networks and the heterogeneity of data limit these advantages. This paper proposes a momentum-accelerated distributed stochastic gradient algorithm, referred to as Exact-Diffusion with Momentum (EDM), which can correct the bias caused by data heterogeneity and introduces the momentum method commonly used in deep learning to accelerate the convergence of the algorithm. We theoretically demonstrate that this algorithm converges to the neighborhood of the optimum sub-linearly irrelevant to data heterogeneity when applied to non-convex objective functions and linearly under the Polyak-Łojasiewicz condition (a weaker assumption than $\mu$-strongly convexity). Finally, we evaluate the performance of the proposed algorithm by simulation, comparing it with a range of existing decentralized optimization algorithms to demonstrate its effectiveness in addressing data heterogeneity and network sparsity.

Title: MotionPCM: Real-Time Motion Synthesis with Phased Consistency Model

• Authors: Lei Jiang, Ye Wei, Hao Ni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19083
• Pdf URL: https://arxiv.org/pdf/2501.19083
• Copy Paste: [[2501.19083]] MotionPCM: Real-Time Motion Synthesis with Phased Consistency Model(https://arxiv.org/abs/2501.19083)
• Keywords: diffusion, generative
• Abstract: Diffusion models have become a popular choice for human motion synthesis due to their powerful generative capabilities. However, their high computational complexity and large sampling steps pose challenges for real-time applications. Fortunately, the Consistency Model (CM) provides a solution to greatly reduce the number of sampling steps from hundreds to a few, typically fewer than four, significantly accelerating the synthesis of diffusion models. However, its application to text-conditioned human motion synthesis in latent space remains challenging. In this paper, we introduce \textbf{MotionPCM}, a phased consistency model-based approach designed to improve the quality and efficiency of real-time motion synthesis in latent space.

Title: Laser: Efficient Language-Guided Segmentation in Neural Radiance Fields

• Authors: Xingyu Miao, Haoran Duan, Yang Bai, Tejal Shah, Jun Song, Yang Long, Rajiv Ranjan, Ling Shao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19084
• Pdf URL: https://arxiv.org/pdf/2501.19084
• Copy Paste: [[2501.19084]] Laser: Efficient Language-Guided Segmentation in Neural Radiance Fields(https://arxiv.org/abs/2501.19084)
• Keywords: segmentation
• Abstract: In this work, we propose a method that leverages CLIP feature distillation, achieving efficient 3D segmentation through language guidance. Unlike previous methods that rely on multi-scale CLIP features and are limited by processing speed and storage requirements, our approach aims to streamline the workflow by directly and effectively distilling dense CLIP features, thereby achieving precise segmentation of 3D scenes using text. To achieve this, we introduce an adapter module and mitigate the noise issue in the dense CLIP feature distillation process through a self-cross-training strategy. Moreover, to enhance the accuracy of segmentation edges, this work presents a low-rank transient query attention mechanism. To ensure the consistency of segmentation for similar colors under different viewpoints, we convert the segmentation task into a classification task through label volume, which significantly improves the consistency of segmentation in color-similar areas. We also propose a simplified text augmentation strategy to alleviate the issue of ambiguity in the correspondence between CLIP features and text. Extensive experimental results show that our method surpasses current state-of-the-art technologies in both training speed and performance. Our code is available on: this https URL.

Title: Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classification

• Authors: Xiangyu Sun, Xiaoguang Zou, Yuanquan Wu, Guotai Wang, Shaoting Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19086
• Pdf URL: https://arxiv.org/pdf/2501.19086
• Copy Paste: [[2501.19086]] Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classification(https://arxiv.org/abs/2501.19086)
• Keywords: fair
• Abstract: X-ray imaging is pivotal in medical diagnostics, offering non-invasive insights into a range of health conditions. Recently, vision-language models, such as the Contrastive Language-Image Pretraining (CLIP) model, have demonstrated potential in improving diagnostic accuracy by leveraging large-scale image-text datasets. However, since CLIP was not initially designed for medical images, several CLIP-like models trained specifically on medical images have been developed. Despite their enhanced performance, issues of fairness - particularly regarding demographic attributes - remain largely unaddressed. In this study, we perform a comprehensive fairness analysis of CLIP-like models applied to X-ray image classification. We assess their performance and fairness across diverse patient demographics and disease categories using zero-shot inference and various fine-tuning techniques, including Linear Probing, Multilayer Perceptron (MLP), Low-Rank Adaptation (LoRA), and full fine-tuning. Our results indicate that while fine-tuning improves model accuracy, fairness concerns persist, highlighting the need for further fairness interventions in these foundational models.

Title: Understanding Oversmoothing in GNNs as Consensus in Opinion Dynamics

• Authors: Keqin Wang, Yulong Yang, Ishan Saha, Christine Allen-Blanchette
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19089
• Pdf URL: https://arxiv.org/pdf/2501.19089
• Copy Paste: [[2501.19089]] Understanding Oversmoothing in GNNs as Consensus in Opinion Dynamics(https://arxiv.org/abs/2501.19089)
• Keywords: attack, robust
• Abstract: In contrast to classes of neural networks where the learned representations become increasingly expressive with network depth, the learned representations in graph neural networks (GNNs), tend to become increasingly similar. This phenomena, known as oversmoothing, is characterized by learned representations that cannot be reliably differentiated leading to reduced predictive performance. In this paper, we propose an analogy between oversmoothing in GNNs and consensus or agreement in opinion dynamics. Through this analogy, we show that the message passing structure of recent continuous-depth GNNs is equivalent to a special case of opinion dynamics (i.e., linear consensus models) which has been theoretically proven to converge to consensus (i.e., oversmoothing) for all inputs. Using the understanding developed through this analogy, we design a new continuous-depth GNN model based on nonlinear opinion dynamics and prove that our model, which we call behavior-inspired message passing neural network (BIMP) circumvents oversmoothing for general inputs. Through extensive experiments, we show that BIMP is robust to oversmoothing and adversarial attack, and consistently outperforms competitive baselines on numerous benchmarks.

Title: Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models

• Authors: Jialin Zhao, Yingtao Zhang, Carlo Vittorio Cannistraci
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19090
• Pdf URL: https://arxiv.org/pdf/2501.19090
• Copy Paste: [[2501.19090]] Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models(https://arxiv.org/abs/2501.19090)
• Keywords: large language model
• Abstract: The rapid growth of Large Language Models has driven demand for effective model compression techniques to reduce memory and computation costs. Low-rank pruning has gained attention for its tensor coherence and GPU compatibility across all densities. However, low-rank pruning has struggled to match the performance of semi-structured pruning, often doubling perplexity (PPL) at similar densities. In this paper, we propose Pivoting Factorization (PIFA), a novel lossless meta low-rank representation that unsupervisedly learns a compact form of any low-rank representation, effectively eliminating redundant information. PIFA identifies pivot rows (linearly independent rows) and expresses non-pivot rows as linear combinations, achieving an additional 24.2\% memory savings and 24.6\% faster inference over low-rank layers at r/d = 0.5, thereby significantly enhancing performance at the same density. To mitigate the performance degradation caused by low-rank pruning, we introduce a novel, retraining-free low-rank reconstruction method that minimizes error accumulation (M). MPIFA, combining M and PIFA into an end-to-end framework, significantly outperforms existing low-rank pruning methods and, for the first time, achieves performance comparable to semi-structured pruning, while surpassing it in GPU efficiency and compatibility.

Title: Improving Low-Resource Sequence Labeling with Knowledge Fusion and Contextual Label Explanations

• Authors: Peichao Lai, Jiaxin Gan, Feiyang Ye, Yilei Wang, Bin Cui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19093
• Pdf URL: https://arxiv.org/pdf/2501.19093
• Copy Paste: [[2501.19093]] Improving Low-Resource Sequence Labeling with Knowledge Fusion and Contextual Label Explanations(https://arxiv.org/abs/2501.19093)
• Keywords: extraction
• Abstract: Sequence labeling remains a significant challenge in low-resource, domain-specific scenarios, particularly for character-dense languages like Chinese. Existing methods primarily focus on enhancing model comprehension and improving data diversity to boost performance. However, these approaches still struggle with inadequate model applicability and semantic distribution biases in domain-specific contexts. To overcome these limitations, we propose a novel framework that combines an LLM-based knowledge enhancement workflow with a span-based Knowledge Fusion for Rich and Efficient Extraction (KnowFREE) model. Our workflow employs explanation prompts to generate precise contextual interpretations of target entities, effectively mitigating semantic biases and enriching the model's contextual understanding. The KnowFREE model further integrates extension label features, enabling efficient nested entity extraction without relying on external knowledge during inference. Experiments on multiple Chinese domain-specific sequence labeling datasets demonstrate that our approach achieves state-of-the-art performance, effectively addressing the challenges posed by low-resource settings.

Title: Ambient Denoising Diffusion Generative Adversarial Networks for Establishing Stochastic Object Models from Noisy Image Data

• Authors: Xichen Xu, Wentao Chen, Weimin Zhou
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.19094
• Pdf URL: https://arxiv.org/pdf/2501.19094
• Copy Paste: [[2501.19094]] Ambient Denoising Diffusion Generative Adversarial Networks for Establishing Stochastic Object Models from Noisy Image Data(https://arxiv.org/abs/2501.19094)
• Keywords: diffusion, generative
• Abstract: It is widely accepted that medical imaging systems should be objectively assessed via task-based image quality (IQ) measures that ideally account for all sources of randomness in the measured image data, including the variation in the ensemble of objects to be imaged. Stochastic object models (SOMs) that can randomly draw samples from the object distribution can be employed to characterize object variability. To establish realistic SOMs for task-based IQ analysis, it is desirable to employ experimental image data. However, experimental image data acquired from medical imaging systems are subject to measurement noise. Previous work investigated the ability of deep generative models (DGMs) that employ an augmented generative adversarial network (GAN), AmbientGAN, for establishing SOMs from noisy measured image data. Recently, denoising diffusion models (DDMs) have emerged as a leading DGM for image synthesis and can produce superior image quality than GANs. However, original DDMs possess a slow image-generation process because of the Gaussian assumption in the denoising steps. More recently, denoising diffusion GAN (DDGAN) was proposed to permit fast image generation while maintain high generated image quality that is comparable to the original DDMs. In this work, we propose an augmented DDGAN architecture, Ambient DDGAN (ADDGAN), for learning SOMs from noisy image data. Numerical studies that consider clinical computed tomography (CT) images and digital breast tomosynthesis (DBT) images are conducted. The ability of the proposed ADDGAN to learn realistic SOMs from noisy image data is demonstrated. It has been shown that the ADDGAN significantly outperforms the advanced AmbientGAN models for synthesizing high resolution medical images with complex textures.

Title: Unraveling Zeroth-Order Optimization through the Lens of Low-Dimensional Structured Perturbations

• Authors: Sihwan Park, Jihun Yun, SungYub Kim, Souvik Kundu, Eunho Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19099
• Pdf URL: https://arxiv.org/pdf/2501.19099
• Copy Paste: [[2501.19099]] Unraveling Zeroth-Order Optimization through the Lens of Low-Dimensional Structured Perturbations(https://arxiv.org/abs/2501.19099)
• Keywords: large language model
• Abstract: Zeroth-order (ZO) optimization has emerged as a promising alternative to gradient-based backpropagation methods, particularly for black-box optimization and large language model (LLM) fine-tuning. However, ZO methods suffer from slow convergence due to high-variance stochastic gradient estimators. While structured perturbations, such as sparsity and low-rank constraints, have been explored to mitigate these issues, their effectiveness remains highly under-explored. In this work, we develop a unified theoretical framework that analyzes both the convergence and generalization properties of ZO optimization under structured perturbations. We show that high dimensionality is the primary bottleneck and introduce the notions of \textit{stable rank} and \textit{effective overlap} to explain how structured perturbations reduce gradient noise and accelerate convergence. Using the uniform stability under our framework, we then provide the first theoretical justification for why these perturbations enhance generalization. Additionally, through empirical analysis, we identify that \textbf{block coordinate descent} (BCD) to be an effective structured perturbation method. Extensive experiments show that, compared to existing alternatives, memory-efficient ZO (MeZO) with BCD (\textit{MeZO-BCD}) can provide improved converge with a faster wall-clock time/iteration by up to $\times\textbf{2.09}$ while yielding similar or better accuracy.

Title: Brain-inspired sparse training enables Transformers and LLMs to perform as fully connected

• Authors: Yingtao Zhang, Jialin Zhao, Wenjing Wu, Ziheng Liao, Umberto Michieli, Carlo Vittorio Cannistraci
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19107
• Pdf URL: https://arxiv.org/pdf/2501.19107
• Copy Paste: [[2501.19107]] Brain-inspired sparse training enables Transformers and LLMs to perform as fully connected(https://arxiv.org/abs/2501.19107)
• Keywords: transformer
• Abstract: This study aims to enlarge our current knowledge on application of brain-inspired network science principles for training artificial neural networks (ANNs) with sparse connectivity. Dynamic sparse training (DST) can reduce the computational demands in ANNs, but faces difficulties to keep peak performance at high sparsity levels. The Cannistraci-Hebb training (CHT) is a brain-inspired method for growing connectivity in DST. CHT leverages a gradient-free, topology-driven link regrowth, which has shown ultra-sparse (1% connectivity or lower) advantage across various tasks compared to fully connected networks. Yet, CHT suffers two main drawbacks: (i) its time complexity is O(Nd^3) - N node network size, d node degree - hence it can apply only to ultra-sparse networks. (ii) it selects top link prediction scores, which is inappropriate for the early training epochs, when the network presents unreliable connections. We propose a GPU-friendly approximation of the CH link predictor, which reduces the computational complexity to O(N^3), enabling a fast implementation of CHT in large-scale models. We introduce the Cannistraci-Hebb training soft rule (CHTs), which adopts a strategy for sampling connections in both link removal and regrowth, balancing the exploration and exploitation of network topology. To improve performance, we integrate CHTs with a sigmoid gradual density decay (CHTss). Empirical results show that, using 1% of connections, CHTs outperforms fully connected networks in MLP on visual classification tasks, compressing some networks to < 30% nodes. Using 5% of the connections, CHTss outperforms fully connected networks in two Transformer-based machine translation tasks. Using 30% of the connections, CHTss achieves superior performance compared to other dynamic sparse training methods in language modeling, and it surpasses the fully connected counterpart in zero-shot evaluations.

Title: Hierarchical Cryptographic Signature Mapping for Ransomware Classification: A Structural Decomposition Approach

• Authors: Dominic Abernethy, Nathaniel Weatherstone, Tristan Ravensdale, Lafedi Svet
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.19120
• Pdf URL: https://arxiv.org/pdf/2501.19120
• Copy Paste: [[2501.19120]] Hierarchical Cryptographic Signature Mapping for Ransomware Classification: A Structural Decomposition Approach(https://arxiv.org/abs/2501.19120)
• Keywords: security, attack
• Abstract: Encryption-based cyber threats continue to evolve, leveraging increasingly sophisticated cryptographic techniques to evade detection and persist within compromised systems. A hierarchical classification framework designed to analyze structural cryptographic properties provides a novel approach to distinguishing malicious encryption from legitimate cryptographic operations. By systematically decomposing encryption workflows into hierarchical layers, the classification method enhances the ability to recognize distinct patterns across diverse threat variants, reducing the dependence on predefined signatures that often fail against rapidly mutating threats. The study examines how cryptographic feature mapping facilitates improved classification accuracy, highlighting the role of entropy, key exchange mechanisms, and algorithmic dependencies in distinguishing harmful encryption activities. Through experimental validation, the framework demonstrated a high degree of precision across multiple attack families, outperforming conventional classification techniques while maintaining computational efficiency suitable for large-scale cybersecurity applications. The layered structural analysis further enhances forensic investigations, enabling security analysts to dissect encryption workflows to trace attack origins and identify commonalities across different campaigns. The methodology strengthens proactive threat mitigation efforts, offering a scalable and adaptable solution that accounts for both known and emerging encryption-based cyber threats. Comparative evaluations illustrate the advantages of structural decomposition in mitigating false positives and negatives, reinforcing the reliability of cryptographic signature classification in real-world security environments.

Title: FedRTS: Federated Robust Pruning via Combinatorial Thompson Sampling

• Authors: Hong Huang, Hai Yang, Yuan Chen, Jiaxun Ye, Dapeng Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19122
• Pdf URL: https://arxiv.org/pdf/2501.19122
• Copy Paste: [[2501.19122]] FedRTS: Federated Robust Pruning via Combinatorial Thompson Sampling(https://arxiv.org/abs/2501.19122)
• Keywords: robust, federate
• Abstract: Federated Learning (FL) enables collaborative model training across distributed clients without data sharing, but its high computational and communication demands strain resource-constrained devices. While existing methods use dynamic pruning to improve efficiency by periodically adjusting sparse model topologies while maintaining sparsity, these approaches suffer from issues such as greedy adjustments, unstable topologies, and communication inefficiency, resulting in less robust models and suboptimal performance under data heterogeneity and partial client availability. To address these challenges, we propose Federated Robust pruning via combinatorial Thompson Sampling (FedRTS), a novel framework designed to develop robust sparse models. FedRTS enhances robustness and performance through its Thompson Sampling-based Adjustment (TSAdj) mechanism, which uses probabilistic decisions informed by stable, farsighted information instead of deterministic decisions reliant on unstable and myopic information in previous methods. Extensive experiments demonstrate that FedRTS achieves state-of-the-art performance in computer vision and natural language processing tasks while reducing communication costs, particularly excelling in scenarios with heterogeneous data distributions and partial client participation. Our codes are available at: this https URL

Title: A theoretical framework for overfitting in energy-based modeling

• Authors: Giovanni Catania, Aurélien Decelle, Cyril Furtlehner, Beatriz Seoane
• Subjects: cs.LG, cond-mat.dis-nn, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2501.19158
• Pdf URL: https://arxiv.org/pdf/2501.19158
• Copy Paste: [[2501.19158]] A theoretical framework for overfitting in energy-based modeling(https://arxiv.org/abs/2501.19158)
• Keywords: generative
• Abstract: We investigate the impact of limited data on training pairwise energy-based models for inverse problems aimed at identifying interaction networks. Utilizing the Gaussian model as testbed, we dissect training trajectories across the eigenbasis of the coupling matrix, exploiting the independent evolution of eigenmodes and revealing that the learning timescales are tied to the spectral decomposition of the empirical covariance matrix. We see that optimal points for early stopping arise from the interplay between these timescales and the initial conditions of training. Moreover, we show that finite data corrections can be accurately modeled through asymptotic random matrix theory calculations and provide the counterpart of generalized cross-validation in the energy based model context. Our analytical framework extends to binary-variable maximum-entropy pairwise models with minimal variations. These findings offer strategies to control overfitting in discrete-variable models through empirical shrinkage corrections, improving the management of overfitting in energy-based generative models.

Title: GDO: Gradual Domain Osmosis

• Authors: Zixi Wang, Yubo Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19159
• Pdf URL: https://arxiv.org/pdf/2501.19159
• Copy Paste: [[2501.19159]] GDO: Gradual Domain Osmosis(https://arxiv.org/abs/2501.19159)
• Keywords: robust
• Abstract: In this paper, we propose a new method called Gradual Domain Osmosis, which aims to solve the problem of smooth knowledge migration from source domain to target domain in Gradual Domain Adaptation (GDA). Traditional Gradual Domain Adaptation methods mitigate domain bias by introducing intermediate domains and self-training strategies, but often face the challenges of inefficient knowledge migration or missing data in intermediate domains. In this paper, we design an optimisation framework based on the hyperparameter $\lambda$ by dynamically balancing the loss weights of the source and target domains, which enables the model to progressively adjust the strength of knowledge migration ($\lambda$ incrementing from 0 to 1) during the training process, thus achieving cross-domain generalisation more efficiently. Specifically, the method incorporates self-training to generate pseudo-labels and iteratively updates the model by minimising a weighted loss function to ensure stability and robustness during progressive adaptation in the intermediate domain. The experimental part validates the effectiveness of the method on rotated MNIST, colour-shifted MNIST, portrait dataset and forest cover type dataset, and the results show that it outperforms existing baseline methods. The paper further analyses the impact of the dynamic tuning strategy of the hyperparameter $\lambda$ on the performance through ablation experiments, confirming the advantages of progressive domain penetration in mitigating the domain bias and enhancing the model generalisation capability. The study provides a theoretical support and practical framework for asymptotic domain adaptation and expands its application potential in dynamic environments.

Title: RMDM: Radio Map Diffusion Model with Physics Informed

• Authors: Haozhe Jia, Wenshuo Chen, Zhihui Huang, Hongru Xiao, Nanqian Jia, Keming Wu, Songning Lai, Yutao Yue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19160
• Pdf URL: https://arxiv.org/pdf/2501.19160
• Copy Paste: [[2501.19160]] RMDM: Radio Map Diffusion Model with Physics Informed(https://arxiv.org/abs/2501.19160)
• Keywords: robust, diffusion
• Abstract: With the rapid development of wireless communication technology, the efficient utilization of spectrum resources, optimization of communication quality, and intelligent communication have become critical. Radio map reconstruction is essential for enabling advanced applications, yet challenges such as complex signal propagation and sparse data hinder accurate reconstruction. To address these issues, we propose the **Radio Map Diffusion Model (RMDM)**, a physics-informed framework that integrates **Physics-Informed Neural Networks (PINNs)** to incorporate constraints like the **Helmholtz equation**. RMDM employs a dual U-Net architecture: the first ensures physical consistency by minimizing PDE residuals, boundary conditions, and source constraints, while the second refines predictions via diffusion-based denoising. By leveraging physical laws, RMDM significantly enhances accuracy, robustness, and generalization. Experiments demonstrate that RMDM outperforms state-of-the-art methods, achieving **NMSE of 0.0031** and **RMSE of 0.0125** under the Static RM (SRM) setting, and **NMSE of 0.0047** and **RMSE of 0.0146** under the Dynamic RM (DRM) setting. These results establish a novel paradigm for integrating physics-informed and data-driven approaches in radio map reconstruction, particularly under sparse data conditions.

Title: Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs

• Authors: Kejia Zhang, Keda Tao, Jiasheng Tang, Huan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19164
• Pdf URL: https://arxiv.org/pdf/2501.19164
• Copy Paste: [[2501.19164]] Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs(https://arxiv.org/abs/2501.19164)
• Keywords: large language model
• Abstract: Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual adversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations.

Title: PSyDUCK: Training-Free Steganography for Latent Diffusion

• Authors: Georgia Channing, Aqib Mahfuz, Mark van der Wilk, Philip Torr, Fabio Pizzati, Christian Schroeder de Witt
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.19172
• Pdf URL: https://arxiv.org/pdf/2501.19172
• Copy Paste: [[2501.19172]] PSyDUCK: Training-Free Steganography for Latent Diffusion(https://arxiv.org/abs/2501.19172)
• Keywords: secure, privacy, diffusion, generative
• Abstract: Recent advances in AI-generated steganography highlight its potential for safeguarding the privacy of vulnerable democratic actors, including aid workers, journalists, and whistleblowers operating in oppressive regimes. In this work, we address current limitations and establish the foundations for large-throughput generative steganography. We introduce a novel approach that enables secure and efficient steganography within latent diffusion models. We show empirically that our methods perform well across a variety of open-source latent diffusion models, particularly in generative image and video tasks.

Title: Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning

• Authors: Xianglin Yang, Gelei Deng, Jieming Shi, Tianwei Zhang, Jin Song Dong
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19180
• Pdf URL: https://arxiv.org/pdf/2501.19180
• Copy Paste: [[2501.19180]] Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning(https://arxiv.org/abs/2501.19180)
• Keywords: defense, attack, large language model
• Abstract: Large language models (LLMs) are vital for a wide range of applications yet remain susceptible to jailbreak threats, which could lead to the generation of inappropriate responses. Conventional defenses, such as refusal and adversarial training, often fail to cover corner cases or rare domains, leaving LLMs still vulnerable to more sophisticated attacks. We propose a novel defense strategy, Safety Chain-of-Thought (SCoT), which harnesses the enhanced \textit{reasoning capabilities} of LLMs for proactive assessment of harmful inputs, rather than simply blocking them. SCoT augments any refusal training datasets to critically analyze the intent behind each request before generating answers. By employing proactive reasoning, SCoT enhances the generalization of LLMs across varied harmful queries and scenarios not covered in the safety alignment corpus. Additionally, it generates detailed refusals specifying the rules violated. Comparative evaluations show that SCoT significantly surpasses existing defenses, reducing vulnerability to out-of-distribution issues and adversarial manipulations while maintaining strong general capabilities.

Title: A Comunication Framework for Compositional Generation

• Authors: Rafael Elberg, Mircea Petrache, Denis Parra
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19182
• Pdf URL: https://arxiv.org/pdf/2501.19182
• Copy Paste: [[2501.19182]] A Comunication Framework for Compositional Generation(https://arxiv.org/abs/2501.19182)
• Keywords: robust, generative
• Abstract: Compositionality and compositional generalization--the ability to understand novel combinations of known concepts--are central characteristics of human language and are hypothesized to be essential for human cognition. In machine learning, the emergence of this property has been studied in a communication game setting, where independent agents (a sender and a receiver) converge to a shared encoding policy from a set of states to a space of discrete messages, where the receiver can correctly reconstruct the states observed by the sender using only the sender's messages. The use of communication games in generation tasks is still largely unexplored, with recent methods for compositional generation focusing mainly on the use of supervised guidance (either through class labels or text). In this work, we take the first steps to fill this gap, and we present a self-supervised generative communication game-based framework for creating compositional encodings in learned representations from pre-trained encoder-decoder models. In an Iterated Learning (IL) protocol involving a sender and a receiver, we apply alternating pressures for compression and diversity of encoded discrete messages, so that the protocol converges to an efficient but unambiguous encoding. Approximate message entropy regularization is used to favor compositional encodings. Our framework is based on rigorous justifications and proofs of defining and balancing the concepts of Eficiency, Unambiguity and Non-Holisticity in encoding. We test our method on the compositional image dataset Shapes3D, demonstrating robust performance in both reconstruction and compositionality metrics, surpassing other tested discrete message frameworks.

Title: Secured Communication Schemes for UAVs in 5G: CRYSTALS-Kyber and IDS

• Authors: Taneya Sharma, Seyed Ahmad Soleymani, Mohammad Shojafar, Rahim Tafazolli
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19191
• Pdf URL: https://arxiv.org/pdf/2501.19191
• Copy Paste: [[2501.19191]] Secured Communication Schemes for UAVs in 5G: CRYSTALS-Kyber and IDS(https://arxiv.org/abs/2501.19191)
• Keywords: secure, security, protect, attack, robust
• Abstract: This paper introduces a secure communication architecture for Unmanned Aerial Vehicles (UAVs) and ground stations in 5G networks, addressing critical challenges in network security. The proposed solution integrates the Advanced Encryption Standard (AES) with Elliptic Curve Cryptography (ECC) and CRYSTALS-Kyber for key encapsulation, offering a hybrid cryptographic approach. By incorporating CRYSTALS-Kyber, the framework mitigates vulnerabilities in ECC against quantum attacks, positioning it as a quantum-resistant alternative. The architecture is based on a server-client model, with UAVs functioning as clients and the ground station acting as the server. The system was rigorously evaluated in both VPN and 5G environments. Experimental results confirm that CRYSTALS-Kyber delivers strong protection against quantum threats with minimal performance overhead, making it highly suitable for UAVs with resource constraints. Moreover, the proposed architecture integrates an Artificial Intelligence (AI)-based Intrusion Detection System (IDS) to further enhance security. In performance evaluations, the IDS demonstrated strong results across multiple models with XGBoost, particularly in more demanding scenarios, outperforming other models with an accuracy of 97.33% and an AUC of 0.94. These findings underscore the potential of combining quantum-resistant encryption mechanisms with AI-driven IDS to create a robust, scalable, and secure communication framework for UAV networks, particularly within the high-performance requirements of 5G environments.

Title: A Variational Perspective on Generative Protein Fitness Optimization

• Authors: Lea Bogensperger, Dominik Narnhofer, Ahmed Allam, Konrad Schindler, Michael Krauthammer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19200
• Pdf URL: https://arxiv.org/pdf/2501.19200
• Copy Paste: [[2501.19200]] A Variational Perspective on Generative Protein Fitness Optimization(https://arxiv.org/abs/2501.19200)
• Keywords: generative
• Abstract: The goal of protein fitness optimization is to discover new protein variants with enhanced fitness for a given use. The vast search space and the sparsely populated fitness landscape, along with the discrete nature of protein sequences, pose significant challenges when trying to determine the gradient towards configurations with higher fitness. We introduce Variational Latent Generative Protein Optimization (VLGPO), a variational perspective on fitness optimization. Our method embeds protein sequences in a continuous latent space to enable efficient sampling from the fitness distribution and combines a (learned) flow matching prior over sequence mutations with a fitness predictor to guide optimization towards sequences with high fitness. VLGPO achieves state-of-the-art results on two different protein benchmarks of varying complexity. Moreover, the variational design with explicit prior and likelihood functions offers a flexible plug-and-play framework that can be easily customized to suit various protein design tasks.

Title: Efficient Reasoning with Hidden Thinking

• Authors: Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, Jiuxiang Gu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19201
• Pdf URL: https://arxiv.org/pdf/2501.19201
• Copy Paste: [[2501.19201]] Efficient Reasoning with Hidden Thinking(https://arxiv.org/abs/2501.19201)
• Keywords: robust, interpretability, large language model
• Abstract: Chain-of-Thought (CoT) reasoning has become a powerful framework for improving complex problem-solving capabilities in Multimodal Large Language Models (MLLMs). However, the verbose nature of textual reasoning introduces significant inefficiencies. In this work, we propose $\textbf{Heima}$ (as hidden llama), an efficient reasoning framework that leverages reasoning CoTs at hidden latent space. We design the Heima Encoder to condense each intermediate CoT into a compact, higher-level hidden representation using a single thinking token, effectively minimizing verbosity and reducing the overall number of tokens required during the reasoning process. Meanwhile, we design corresponding Heima Decoder with traditional Large Language Models (LLMs) to adaptively interpret the hidden representations into variable-length textual sequence, reconstructing reasoning processes that closely resemble the original CoTs. Experimental results across diverse reasoning MLLM benchmarks demonstrate that Heima model achieves higher generation efficiency while maintaining or even better zero-shot task accuracy. Moreover, the effective reconstruction of multimodal reasoning processes with Heima Decoder validates both the robustness and interpretability of our approach.

Title: Improving the Robustness of Representation Misdirection for Large Language Model Unlearning

• Authors: Dang Huu-Tien, Hoang Thanh-Tung, Le-Minh Nguyen, Naoya Inoue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19202
• Pdf URL: https://arxiv.org/pdf/2501.19202
• Copy Paste: [[2501.19202]] Improving the Robustness of Representation Misdirection for Large Language Model Unlearning(https://arxiv.org/abs/2501.19202)
• Keywords: defense, attack, robust, large language model
• Abstract: Representation Misdirection (RM) and variants are established large language model (LLM) unlearning methods with state-of-the-art performance. In this paper, we show that RM methods inherently reduce models' robustness, causing them to misbehave even when a single non-adversarial forget-token is in the retain-query. Toward understanding underlying causes, we reframe the unlearning process as backdoor attacks and defenses: forget-tokens act as backdoor triggers that, when activated in retain-queries, cause disruptions in RM models' behaviors, similar to successful backdoor attacks. To mitigate this vulnerability, we propose Random Noise Augmentation -- a model and method agnostic approach with theoretical guarantees for improving the robustness of RM methods. Extensive experiments demonstrate that RNA significantly improves the robustness of RM models while enhancing the unlearning performances.

Title: RIGNO: A Graph-based framework for robust and accurate operator learning for PDEs on arbitrary domains

• Authors: Sepehr Mousavi, Shizheng Wen, Levi Lingsch, Maximilian Herde, Bogdan Raonić, Siddhartha Mishra
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19205
• Pdf URL: https://arxiv.org/pdf/2501.19205
• Copy Paste: [[2501.19205]] RIGNO: A Graph-based framework for robust and accurate operator learning for PDEs on arbitrary domains(https://arxiv.org/abs/2501.19205)
• Keywords: robust
• Abstract: Learning the solution operators of PDEs on arbitrary domains is challenging due to the diversity of possible domain shapes, in addition to the often intricate underlying physics. We propose an end-to-end graph neural network (GNN) based neural operator to learn PDE solution operators from data on point clouds in arbitrary domains. Our multi-scale model maps data between input/output point clouds by passing it through a downsampled regional mesh. Many novel elements are also incorporated to ensure resolution invariance and temporal continuity. Our model, termed RIGNO, is tested on a challenging suite of benchmarks, composed of various time-dependent and steady PDEs defined on a diverse set of domains. We demonstrate that RIGNO is significantly more accurate than neural operator baselines and robustly generalizes to unseen spatial resolutions and time instances.

Title: Strassen Attention: Unlocking Compositional Abilities in Transformers Based on a New Lower Bound Method

• Authors: Alexander Kozachinskiy, Felipe Urrutia, Hector Jimenez, Tomasz Steifer, Germán Pizarro, Matías Fuentes, Francisco Meza, Cristian Buc, Cristóbal Rojas
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19215
• Pdf URL: https://arxiv.org/pdf/2501.19215
• Copy Paste: [[2501.19215]] Strassen Attention: Unlocking Compositional Abilities in Transformers Based on a New Lower Bound Method(https://arxiv.org/abs/2501.19215)
• Keywords: transformer
• Abstract: We propose a novel method to evaluate the theoretical limits of Transformers, allowing us to prove the first lower bounds against one-layer softmax Transformers with infinite precision. We establish those bounds for three tasks that require advanced reasoning. The first task, Match3 (Sanford et al., 2023), requires looking at all triples of positions. The second and third tasks address compositionality-based reasoning: one is composition of functions (Peng et al., 2024) and the other is composition of binary relations. We formally prove the inability of one-layer softmax Transformers to solve any of these tasks. In an attempt to overcome these limitations, we introduce Strassen attention and prove that with this mechanism a one-layer Transformer can in principle solve all these tasks. We also show that it enjoys sub-cubic running-time complexity, making it more scalable than similar previously proposed mechanisms, such as higher-order attention (Sanford et al., 2023). To complement our theoretical findings, we experimentally studied Strassen attention and compared it against standard (Vaswani et al, 2017), higher-order attention (Sanford et al., 2023) and triangular attention (Bergen et al. 2021). Our results help to disentangle all these attention mechanisms, highlighting their strengths and limitations. In particular, Strassen attention outperforms standard attention significantly on all the tasks. Altogether, understanding the theoretical limitations can guide research towards scalable attention mechanisms that improve the reasoning abilities of Transformers.

Title: \underline{E2}Former: A Linear-time \underline{E}fficient and \underline{E}quivariant Trans\underline{former} for Scalable Molecular Modeling

• Authors: Yunyang Li, Lin Huang, Zhihao Ding, Chu Wang, Xinran Wei, Han Yang, Zun Wang, Chang Liu, Yu Shi, Peiran Jin, Jia Zhang, Mark Gerstein, Tao Qin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19216
• Pdf URL: https://arxiv.org/pdf/2501.19216
• Copy Paste: [[2501.19216]] \underline{E2}Former: A Linear-time \underline{E}fficient and \underline{E}quivariant Trans\underline{former} for Scalable Molecular Modeling(https://arxiv.org/abs/2501.19216)
• Keywords: transformer
• Abstract: Equivariant Graph Neural Networks (EGNNs) have demonstrated significant success in modeling microscale systems, including those in chemistry, biology and materials science. However, EGNNs face substantial computational challenges due to the high cost of constructing edge features via spherical tensor products, making them impractical for large-scale systems. To address this limitation, we introduce E2Former, an equivariant and efficient transformer architecture that incorporates the Wigner $6j$ convolution (Wigner $6j$ Conv). By shifting the computational burden from edges to nodes, the Wigner $6j$ Conv reduces the complexity from $O(|\mathcal{E}|)$ to $ O(| \mathcal{V}|)$ while preserving both the model's expressive power and rotational equivariance. We show that this approach achieves a 7x-30x speedup compared to conventional $\mathrm{SO}(3)$ convolutions. Furthermore, our empirical results demonstrate that the derived E2Former mitigates the computational challenges of existing approaches without compromising the ability to capture detailed geometric information. This development could suggest a promising direction for scalable and efficient molecular modeling.

Title: Through the Looking Glass: LLM-Based Analysis of AR/VR Android Applications Privacy Policies

• Authors: Abdulaziz Alghamdi, David Mohaisen
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19223
• Pdf URL: https://arxiv.org/pdf/2501.19223
• Copy Paste: [[2501.19223]] Through the Looking Glass: LLM-Based Analysis of AR/VR Android Applications Privacy Policies(https://arxiv.org/abs/2501.19223)
• Keywords: privacy
• Abstract: \begin{abstract} This paper comprehensively analyzes privacy policies in AR/VR applications, leveraging BERT, a state-of-the-art text classification model, to evaluate the clarity and thoroughness of these policies. By comparing the privacy policies of AR/VR applications with those of free and premium websites, this study provides a broad perspective on the current state of privacy practices within the AR/VR industry. Our findings indicate that AR/VR applications generally offer a higher percentage of positive segments than free content but lower than premium websites. The analysis of highlighted segments and words revealed that AR/VR applications strategically emphasize critical privacy practices and key terms. This enhances privacy policies' clarity and effectiveness.

Title: Integrating Semi-Supervised and Active Learning for Semantic Segmentation

• Authors: Wanli Ma, Oktay Karakus, Paul L. Rosin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19227
• Pdf URL: https://arxiv.org/pdf/2501.19227
• Copy Paste: [[2501.19227]] Integrating Semi-Supervised and Active Learning for Semantic Segmentation(https://arxiv.org/abs/2501.19227)
• Keywords: segmentation
• Abstract: In this paper, we propose a novel active learning approach integrated with an improved semi-supervised learning framework to reduce the cost of manual annotation and enhance model performance. Our proposed approach effectively leverages both the labelled data selected through active learning and the unlabelled data excluded from the selection process. The proposed active learning approach pinpoints areas where the pseudo-labels are likely to be inaccurate. Then, an automatic and efficient pseudo-label auto-refinement (PLAR) module is proposed to correct pixels with potentially erroneous pseudo-labels by comparing their feature representations with those of labelled regions. This approach operates without increasing the labelling budget and is based on the cluster assumption, which states that pixels belonging to the same class should exhibit similar representations in feature space. Furthermore, manual labelling is only applied to the most difficult and uncertain areas in unlabelled data, where insufficient information prevents the PLAR module from making a decision. We evaluated the proposed hybrid semi-supervised active learning framework on two benchmark datasets, one from natural and the other from remote sensing imagery domains. In both cases, it outperformed state-of-the-art methods in the semantic segmentation task.

Title: Accelerating Diffusion Transformer via Error-Optimized Cache

• Authors: Junxiang Qiu, Shuo Wang, Jinda Lu, Lin Liu, Houcheng Jiang, Yanbin Hao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19243
• Pdf URL: https://arxiv.org/pdf/2501.19243
• Copy Paste: [[2501.19243]] Accelerating Diffusion Transformer via Error-Optimized Cache(https://arxiv.org/abs/2501.19243)
• Keywords: extraction, diffusion, transformer
• Abstract: Diffusion Transformer (DiT) is a crucial method for content generation. However, it needs a lot of time to sample. Many studies have attempted to use caching to reduce the time consumption of sampling. Existing caching methods accelerate generation by reusing DiT features from the previous time step and skipping calculations in the next, but they tend to locate and cache low-error modules without focusing on reducing caching-induced errors, resulting in a sharp decline in generated content quality when increasing caching intensity. To solve this problem, we propose the Error-Optimized Cache (EOC). This method introduces three key improvements: (1) Prior knowledge extraction: Extract and process the caching differences; (2) A judgment method for cache optimization: Determine whether certain caching steps need to be optimized; (3) Cache optimization: reduce caching errors. Experiments show that this algorithm significantly reduces the error accumulation caused by caching (especially over-caching). On the ImageNet dataset, without significantly increasing the computational burden, this method improves the quality of the generated images under the over-caching, rule-based, and training-based methods. Specifically, the Fréchet Inception Distance (FID) values are improved as follows: from 6.857 to 5.821, from 3.870 to 3.692 and form 3.539 to 3.451 respectively.

Title: Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search

• Authors: Yuta Oshima, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19252
• Pdf URL: https://arxiv.org/pdf/2501.19252
• Copy Paste: [[2501.19252]] Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search(https://arxiv.org/abs/2501.19252)
• Keywords: diffusion
• Abstract: The remarkable progress in text-to-video diffusion models enables photorealistic generations, although the contents of the generated video often include unnatural movement or deformation, reverse playback, and motionless scenes. Recently, an alignment problem has attracted huge attention, where we steer the output of diffusion models based on some quantity on the goodness of the content. Because there is a large room for improvement of perceptual quality along the frame direction, we should address which metrics we should optimize and how we can optimize them in the video generation. In this paper, we propose diffusion latent beam search with lookahead estimator, which can select better diffusion latent to maximize a given alignment reward, at inference time. We then point out that the improvement of perceptual video quality considering the alignment to prompts requires reward calibration by weighting existing metrics. When evaluating outputs by using vision language models as a proxy of humans, many previous metrics to quantify the naturalness of video do not always correlate with evaluation and also depend on the degree of dynamic descriptions in evaluation prompts. We demonstrate that our method improves the perceptual quality based on the calibrated reward, without model parameter update, and outputs the best generation compared to greedy search and best-of-N sampling. We provide practical guidelines on which axes, among search budget, lookahead steps for reward estimate, and denoising steps, in the reverse diffusion process, we should allocate the inference-time computation.

Title: ContextFormer: Redefining Efficiency in Semantic Segmentation

• Authors: Mian Muhammad Naeem Abid, Nancy Mehta, Zongwei Wu, Fayaz Ali Dharejo, Radu Timofte
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19255
• Pdf URL: https://arxiv.org/pdf/2501.19255
• Copy Paste: [[2501.19255]] ContextFormer: Redefining Efficiency in Semantic Segmentation(https://arxiv.org/abs/2501.19255)
• Keywords: robust, extraction, transformer, segmentation
• Abstract: Semantic segmentation assigns labels to pixels in images, a critical yet challenging task in computer vision. Convolutional methods, although capturing local dependencies well, struggle with long-range relationships. Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands, especially for high-resolution inputs. Most research optimizes the encoder architecture, leaving the bottleneck underexplored - a key area for enhancing performance and efficiency. We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation. The framework's efficiency is driven by three synergistic modules: the Token Pyramid Extraction Module (TPEM) for hierarchical multi-scale representation, the Transformer and Modulating DepthwiseConv (Trans-MDC) block for dynamic scale-aware feature modeling, and the Feature Merging Module (FMM) for robust integration with enhanced spatial and contextual consistency. Extensive experiments on ADE20K, Pascal Context, CityScapes, and COCO-Stuff datasets show ContextFormer significantly outperforms existing models, achieving state-of-the-art mIoU scores, setting a new benchmark for efficiency and performance. The codes will be made publicly available.

Title: Medical Semantic Segmentation with Diffusion Pretrain

• Authors: David Li, Anvar Kurmukov, Mikhail Goncharov, Roman Sokolov, Mikhail Belyaev
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19265
• Pdf URL: https://arxiv.org/pdf/2501.19265
• Copy Paste: [[2501.19265]] Medical Semantic Segmentation with Diffusion Pretrain(https://arxiv.org/abs/2501.19265)
• Keywords: robust, diffusion, transformer, segmentation
• Abstract: Recent advances in deep learning have shown that learning robust feature representations is critical for the success of many computer vision tasks, including medical image segmentation. In particular, both transformer and convolutional-based architectures have benefit from leveraging pretext tasks for pretraining. However, the adoption of pretext tasks in 3D medical imaging has been less explored and remains a challenge, especially in the context of learning generalizable feature representations. We propose a novel pretraining strategy using diffusion models with anatomical guidance, tailored to the intricacies of 3D medical image data. We introduce an auxiliary diffusion process to pretrain a model that produce generalizable feature representations, useful for a variety of downstream segmentation tasks. We employ an additional model that predicts 3D universal body-part coordinates, providing guidance during the diffusion process and improving spatial awareness in generated representations. This approach not only aids in resolving localization inaccuracies but also enriches the model's ability to understand complex anatomical structures. Empirical validation on a 13-class organ segmentation task demonstrate the effectiveness of our pretraining technique. It surpasses existing restorative pretraining methods in 3D medical image segmentation by $7.5\%$, and is competitive with the state-of-the-art contrastive pretraining approach, achieving an average Dice coefficient of 67.8 in a non-linear evaluation scenario.

Title: Pheromone-based Learning of Optimal Reasoning Paths

• Authors: Anirudh Chari, Aditya Tiwari, Richard Lian, Suraj Reddy, Brian Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19278
• Pdf URL: https://arxiv.org/pdf/2501.19278
• Copy Paste: [[2501.19278]] Pheromone-based Learning of Optimal Reasoning Paths(https://arxiv.org/abs/2501.19278)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities through chain-of-thought prompting, yet discovering effective reasoning methods for complex problems remains challenging due to the vast space of possible intermediate steps. We introduce Ant Colony Optimization-guided Tree of Thought (ACO-ToT), a novel algorithm that combines ACO with LLMs to discover optimal reasoning paths for complex problems efficiently. Drawing inspiration from Hebbian learning in neurological systems, our method employs a collection of distinctly fine-tuned LLM "ants" to traverse and lay pheromone trails through a centralized tree of thought, with each ant's movement governed by a weighted combination of existing pheromone trails and its own specialized expertise. The algorithm evaluates complete reasoning paths using a mixture-of-experts-based scoring function, with pheromones reinforcing productive reasoning paths across iterations. Experiments on three challenging reasoning tasks (GSM8K, ARC-Challenge, and MATH) demonstrate that ACO-ToT performs significantly better than existing chain-of-thought optimization approaches, suggesting that incorporating biologically inspired collective search mechanisms into LLM inference can substantially enhance reasoning capabilities.

Title: S-VOTE: Similarity-based Voting for Client Selection in Decentralized Federated Learning

• Authors: Pedro Miguel Sánchez Sánchez, Enrique Tomás Martínez Beltrán, Chao Feng, Gérôme Bovet, Gregorio Martínez Pérez, Alberto Huertas Celdrán
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.19279
• Pdf URL: https://arxiv.org/pdf/2501.19279
• Copy Paste: [[2501.19279]] S-VOTE: Similarity-based Voting for Client Selection in Decentralized Federated Learning(https://arxiv.org/abs/2501.19279)
• Keywords: privacy, federate
• Abstract: Decentralized Federated Learning (DFL) enables collaborative, privacy-preserving model training without relying on a central server. This decentralized approach reduces bottlenecks and eliminates single points of failure, enhancing scalability and resilience. However, DFL also introduces challenges such as suboptimal models with non-IID data distributions, increased communication overhead, and resource usage. Thus, this work proposes S-VOTE, a voting-based client selection mechanism that optimizes resource usage and enhances model performance in federations with non-IID data conditions. S-VOTE considers an adaptive strategy for spontaneous local training that addresses participation imbalance, allowing underutilized clients to contribute without significantly increasing resource costs. Extensive experiments on benchmark datasets demonstrate the S-VOTE effectiveness. More in detail, it achieves lower communication costs by up to 21%, 4-6% faster convergence, and improves local performance by 9-17% compared to baseline methods in some configurations, all while achieving a 14-24% energy consumption reduction. These results highlight the potential of S-VOTE to address DFL challenges in heterogeneous environments.

Title: Application of Generative Adversarial Network (GAN) for Synthetic Training Data Creation to improve performance of ANN Classifier for extracting Built-Up pixels from Landsat Satellite Imagery

• Authors: Amritendu Mukherjee, Dipanwita Sinha Mukherjee, Parthasarathy Ramachandran
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19283
• Pdf URL: https://arxiv.org/pdf/2501.19283
• Copy Paste: [[2501.19283]] Application of Generative Adversarial Network (GAN) for Synthetic Training Data Creation to improve performance of ANN Classifier for extracting Built-Up pixels from Landsat Satellite Imagery(https://arxiv.org/abs/2501.19283)
• Keywords: generative
• Abstract: Training a neural network for pixel based classification task using low resolution Landsat images is difficult as the size of the training data is usually small due to less number of available pixels that represent a single class without any mixing with other classes. Due to this scarcity of training data, neural network may not be able to attain expected level of accuracy. This limitation could be overcome using a generative network that aims to generate synthetic data having the same distribution as the sample data with which it is trained. In this work, we have proposed a methodology for improving the performance of ANN classifier to identify built-up pixels in the Landsat$7$ image with the help of developing a simple GAN architecture that could generate synthetic training pixels when trained using original set of sample built-up pixels. To ensure that the marginal and joint distributions of all the bands corresponding to the generated and original set of pixels are indistinguishable, non-parametric Kolmogorov Smirnov Test and Ball Divergence based Equality of Distributions Test have been performed respectively. It has been observed that the overall accuracy and kappa coefficient of the ANN model for built-up classification have continuously improved from $0.9331$ to $0.9983$ and $0.8277$ to $0.9958$ respectively, with the inclusion of generated sets of built-up pixels to the original one.

Title: Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs

• Authors: James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19287
• Pdf URL: https://arxiv.org/pdf/2501.19287
• Copy Paste: [[2501.19287]] Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs(https://arxiv.org/abs/2501.19287)
• Keywords: privacy
• Abstract: In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. Differential Privacy (DP), a widely adopted privacy safeguard, has emerged to mitigate this privacy leakage, with recent work demonstrating strong privacy-utility tradeoffs in classification tasks for ICL. However, generation tasks for ICL are challenging due to the high-dimensional output space of open-ended generation. To this end, we propose $\texttt{dps-mozo}$, Differentially Private Sampling by Mixing One-shot with Zero-shot Outputs, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output. This mixing effectively reduces the amount of information that can be leaked by each demonstration. By utilizing the inherent randomness in sampling from the mixed distributions, we can achieve DP without adding noise, thereby improving the privacy-utility tradeoff. Our experimental evaluations show $\texttt{dps-mozo}$ can achieve a strong privacy guarantee, $\epsilon=2$, with minimal utility degradation compared to non-private few-shot learning, $\textbf{0.3}$% ROUGE-L F1 score decrease on the SAMSum dataset with Gemma 2 2B.

Title: Noninterference Analysis of Irreversible or Reversible Systems with Nondeterminism and Probabilities

• Authors: Andrea Esposito, Alessandro Aldini, Marco Bernardo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.19290
• Pdf URL: https://arxiv.org/pdf/2501.19290
• Copy Paste: [[2501.19290]] Noninterference Analysis of Irreversible or Reversible Systems with Nondeterminism and Probabilities(https://arxiv.org/abs/2501.19290)
• Keywords: secure, security, generative
• Abstract: Noninterference theory supports the analysis of secure computations in multi-level security systems. Classical equivalence-based approaches to noninterference mainly rely on bisimilarity. In a nondeterministic setting, assessing noninterference through weak bisimilarity is adequate for irreversible systems, whereas for reversible ones branching bisimilarity has been recently proven to be more appropriate. In this paper we address the same two families of systems, with the difference that probabilities come into play in addition to nondeterminism. For irreversible systems we extend the results of Aldini, Bravetti, and Gorrieri developed in a generative-reactive probabilistic setting, while for reversible systems we extend the results of Esposito, Aldini, Bernardo, and Rossi developed in a purely nondeterministic setting. We recast noninterference properties by adopting probabilistic variants of weak and branching bisimilarities for irreversible and reversible systems respectively. Then we investigate a taxonomy of those properties as well as their preservation and compositionality aspects, along with a comparison with the nondeterministic taxonomy. The adequacy of the extended noninterference theory is illustrated via a probabilistic smart contract example.

Title: Offline Learning for Combinatorial Multi-armed Bandits

• Authors: Xutong Liu, Xiangxiang Dai, Jinhang Zuo, Siwei Wang, Carlee-Joe Wong, John C.S. Lui, Wei Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19300
• Pdf URL: https://arxiv.org/pdf/2501.19300
• Copy Paste: [[2501.19300]] Offline Learning for Combinatorial Multi-armed Bandits(https://arxiv.org/abs/2501.19300)
• Keywords: large language model
• Abstract: The combinatorial multi-armed bandit (CMAB) is a fundamental sequential decision-making framework, extensively studied over the past decade. However, existing work primarily focuses on the online setting, overlooking the substantial costs of online interactions and the readily available offline datasets. To overcome these limitations, we introduce Off-CMAB, the first offline learning framework for CMAB. Central to our framework is the combinatorial lower confidence bound (CLCB) algorithm, which combines pessimistic reward estimations with combinatorial solvers. To characterize the quality of offline datasets, we propose two novel data coverage conditions and prove that, under these conditions, CLCB achieves a near-optimal suboptimality gap, matching the theoretical lower bound up to a logarithmic factor. We validate Off-CMAB through practical applications, including learning to rank, large language model (LLM) caching, and social influence maximization, showing its ability to handle nonlinear reward functions, general feedback models, and out-of-distribution action samples that excludes optimal or even feasible actions. Extensive experiments on synthetic and real-world datasets further highlight the superior performance of CLCB.

Title: Beyond checkmate: exploring the creative chokepoints in AI text

• Authors: Nafis Irtiza Tripto, Saranya Venkatraman, Mahjabin Nahar, Dongwon Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19301
• Pdf URL: https://arxiv.org/pdf/2501.19301
• Copy Paste: [[2501.19301]] Beyond checkmate: exploring the creative chokepoints in AI text(https://arxiv.org/abs/2501.19301)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) and Artificial Intelligence (AI), unlocking unprecedented capabilities. This rapid advancement has spurred research into various aspects of LLMs, their text generation & reasoning capability, and potential misuse, fueling the necessity for robust detection methods. While numerous prior research has focused on detecting LLM-generated text (AI text) and thus checkmating them, our study investigates a relatively unexplored territory: portraying the nuanced distinctions between human and AI texts across text segments. Whether LLMs struggle with or excel at incorporating linguistic ingenuity across different text segments carries substantial implications for determining their potential as effective creative assistants to humans. Through an analogy with the structure of chess games-comprising opening, middle, and end games-we analyze text segments (introduction, body, and conclusion) to determine where the most significant distinctions between human and AI texts exist. While AI texts can approximate the body segment better due to its increased length, a closer examination reveals a pronounced disparity, highlighting the importance of this segment in AI text detection. Additionally, human texts exhibit higher cross-segment differences compared to AI texts. Overall, our research can shed light on the intricacies of human-AI text distinctions, offering novel insights for text detection and understanding.

Title: Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment

• Authors: Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, Jonas Kohler
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19309
• Pdf URL: https://arxiv.org/pdf/2501.19309
• Copy Paste: [[2501.19309]] Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment(https://arxiv.org/abs/2501.19309)
• Keywords: large language model
• Abstract: The performance of large language models (LLMs) is closely linked to their underlying size, leading to ever-growing networks and hence slower inference. Speculative decoding has been proposed as a technique to accelerate autoregressive generation, leveraging a fast draft model to propose candidate tokens, which are then verified in parallel based on their likelihood under the target model. While this approach guarantees to reproduce the target output, it incurs a substantial penalty: many high-quality draft tokens are rejected, even when they represent objectively valid continuations. Indeed, we show that even powerful draft models such as GPT-4o, as well as human text cannot achieve high acceptance rates under the standard verification scheme. This severely limits the speedup potential of current speculative decoding methods, as an early rejection becomes overwhelmingly likely when solely relying on alignment of draft and target. We thus ask the following question: Can we adapt verification to recognize correct, but non-aligned replies? To this end, we draw inspiration from the LLM-as-a-judge framework, which demonstrated that LLMs are able to rate answers in a versatile way. We carefully design a dataset to elicit the same capability in the target model by training a compact module on top of the embeddings to produce ``judgements" of the current continuation. We showcase our strategy on the Llama-3.1 family, where our 8b/405B-Judge achieves a speedup of 9x over Llama-405B, while maintaining its quality on a large range of benchmarks. These benefits remain present even in optimized inference frameworks, where our method reaches up to 141 tokens/s for 8B/70B-Judge and 129 tokens/s for 8B/405B on 2 and 8 H100s respectively.

Title: An Efficient Approach for Machine Translation on Low-resource Languages: A Case Study in Vietnamese-Chinese

• Authors: Tran Ngoc Son, Nguyen Anh Tu, Nguyen Minh Tri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19314
• Pdf URL: https://arxiv.org/pdf/2501.19314
• Copy Paste: [[2501.19314]] An Efficient Approach for Machine Translation on Low-resource Languages: A Case Study in Vietnamese-Chinese(https://arxiv.org/abs/2501.19314)
• Keywords: transformer
• Abstract: Despite the rise of recent neural networks in machine translation, those networks do not work well if the training data is insufficient. In this paper, we proposed an approach for machine translation in low-resource languages such as Vietnamese-Chinese. Our proposed method leveraged the power of the multilingual pre-trained language model (mBART) and both Vietnamese and Chinese monolingual corpus. Firstly, we built an early bird machine translation model using the bilingual training dataset. Secondly, we used TF-IDF technique to select sentences from the monolingual corpus which are the most related to domains of the parallel dataset. Finally, the first model was used to synthesize the augmented training data from the selected monolingual corpus for the translation model. Our proposed scheme showed that it outperformed 8% compared to the transformer model. The augmented dataset also pushed the model performance.

Title: Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution

• Authors: Tatiana Anikina, Arne Binder, David Harbecke, Stalin Varanasi, Leonhard Hennig, Simon Ostermann, Sebastian Möller, Josef van Genabith
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19316
• Pdf URL: https://arxiv.org/pdf/2501.19316
• Copy Paste: [[2501.19316]] Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution(https://arxiv.org/abs/2501.19316)
• Keywords: extraction
• Abstract: In this work, we reimagine classical probing to evaluate knowledge transfer from simple source to more complex target tasks. Instead of probing frozen representations from a complex source task on diverse simple target probing tasks (as usually done in probing), we explore the effectiveness of embeddings from multiple simple source tasks on a single target task. We select coreference resolution, a linguistically complex problem requiring contextual understanding, as focus target task, and test the usefulness of embeddings from comparably simpler tasks tasks such as paraphrase detection, named entity recognition, and relation extraction. Through systematic experiments, we evaluate the impact of individual and combined task embeddings. Our findings reveal that task embeddings vary significantly in utility for coreference resolution, with semantic similarity tasks (e.g., paraphrase detection) proving most beneficial. Additionally, representations from intermediate layers of fine-tuned models often outperform those from final layers. Combining embeddings from multiple tasks consistently improves performance, with attention-based aggregation yielding substantial gains. These insights shed light on relationships between task-specific representations and their adaptability to complex downstream tasks, encouraging further exploration of embedding-level task transfer.

Title: LLM-based Affective Text Generation Quality Based on Different Quantization Values

• Authors: Yarik Menchaca Resendiz, Roman Klinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.19317
• Pdf URL: https://arxiv.org/pdf/2501.19317
• Copy Paste: [[2501.19317]] LLM-based Affective Text Generation Quality Based on Different Quantization Values(https://arxiv.org/abs/2501.19317)
• Keywords: large language model
• Abstract: Large language models exhibit a remarkable capacity in language generation and comprehension. These advances enable AI systems to produce more human-like and emotionally engaging text. However, these models rely on a large number of parameters, requiring significant computational resources for training and inference. In some scenarios, accessing these resources can be challenging (e.g., budget or hardware limitations). Techniques like reducing precision bits can make models more memory-efficient, reducing the computational resources needed, at the cost of reduced accuracy. This paper addresses the trade-off between different quantization values, GPU RAM utilization, and text quality in affective text generation (e.g., "I really enjoy running in the snow-covered forest"). To evaluate, we use an emotion classifier and ten seed prompts to generate affective text. We test three setups of precision bits (8, 16, and 32) across five open-weight language models from two different families. Our findings demonstrate that bit reductions lead to memory savings, achieving a reduction of 76%. However, this optimization comes with a trade-off, leading to a decrease of up to 10 pp in F1 score for larger models and an increase of 10 pp for smaller models, along with roughly double the inference time. In terms of text quality, larger models at lower quantization levels generally outperform smaller, higher-precision models -- while requiring similar memory.

Title: Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping

• Authors: Yiming Huang, Beilei Cui, Long Bai, Zhen Chen, Jinlin Wu, Zhen Li, Hongbin Liu, Hongliang Ren
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.19319
• Pdf URL: https://arxiv.org/pdf/2501.19319
• Copy Paste: [[2501.19319]] Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping(https://arxiv.org/abs/2501.19319)
• Keywords: robust
• Abstract: Simultaneous Localization and Mapping (SLAM) is essential for precise surgical interventions and robotic tasks in minimally invasive procedures. While recent advancements in 3D Gaussian Splatting (3DGS) have improved SLAM with high-quality novel view synthesis and fast rendering, these systems struggle with accurate depth and surface reconstruction due to multi-view inconsistencies. Simply incorporating SLAM and 3DGS leads to mismatches between the reconstructed frames. In this work, we present Endo-2DTAM, a real-time endoscopic SLAM system with 2D Gaussian Splatting (2DGS) to address these challenges. Endo-2DTAM incorporates a surface normal-aware pipeline, which consists of tracking, mapping, and bundle adjustment modules for geometrically accurate reconstruction. Our robust tracking module combines point-to-point and point-to-plane distance metrics, while the mapping module utilizes normal consistency and depth distortion to enhance surface reconstruction quality. We also introduce a pose-consistent strategy for efficient and geometrically coherent keyframe sampling. Extensive experiments on public endoscopic datasets demonstrate that Endo-2DTAM achieves an RMSE of $1.87\pm 0.63$ mm for depth reconstruction of surgical scenes while maintaining computationally efficient tracking, high-quality visual appearance, and real-time rendering. Our code will be released at this http URL.

Title: Reward-Guided Speculative Decoding for Efficient LLM Reasoning

• Authors: Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, Caiming Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19324
• Pdf URL: https://arxiv.org/pdf/2501.19324
• Copy Paste: [[2501.19324]] Reward-Guided Speculative Decoding for Efficient LLM Reasoning(https://arxiv.org/abs/2501.19324)
• Keywords: robust, large language model
• Abstract: We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs). RSD synergistically combines a lightweight draft model with a more powerful target model, incorporating a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness. RSD employs a process reward model to evaluate intermediate decoding steps and dynamically decide whether to invoke the target model, optimizing the trade-off between computational cost and output quality. We theoretically demonstrate that a threshold-based mixture strategy achieves an optimal balance between resource utilization and performance. Extensive evaluations on challenging reasoning benchmarks, including Olympiad-level tasks, show that RSD delivers significant efficiency gains against decoding with the target model only (up to 4.4x fewer FLOPs), while achieving significant better accuracy than parallel decoding method on average (up to +3.5). These results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios.

Title: A Generic Hybrid Framework for 2D Visual Reconstruction

• Authors: Daniel Rika, Dror Sholomon, Eli David, Alexandre Pais, Nathan S. Netanyahu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19325
• Pdf URL: https://arxiv.org/pdf/2501.19325
• Copy Paste: [[2501.19325]] A Generic Hybrid Framework for 2D Visual Reconstruction(https://arxiv.org/abs/2501.19325)
• Keywords: robust
• Abstract: This paper presents a versatile hybrid framework for addressing 2D real-world reconstruction tasks formulated as jigsaw puzzle problems (JPPs) with square, non-overlapping pieces. Our approach integrates a deep learning (DL)-based compatibility measure (CM) model that evaluates pairs of puzzle pieces holistically, rather than focusing solely on their adjacent edges as traditionally done. This DL-based CM is paired with an optimized genetic algorithm (GA)-based solver, which iteratively searches for a global optimal arrangement using the pairwise CM scores of the puzzle pieces. Extensive experimental results highlight the framework's adaptability and robustness across multiple real-world domains. Notably, our unique hybrid methodology achieves state-of-the-art (SOTA) results in reconstructing Portuguese tile panels and large degraded puzzles with eroded boundaries.

Title: Let Human Sketches Help: Empowering Challenging Image Segmentation Task with Freehand Sketches

• Authors: Ying Zang, Runlong Cao, Jianqi Zhang, Yidong Han, Ziyue Cao, Wenjun Hu, Didi Zhu, Lanyun Zhu, Zejian Li, Deyi Ji, Tianrun Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19329
• Pdf URL: https://arxiv.org/pdf/2501.19329
• Copy Paste: [[2501.19329]] Let Human Sketches Help: Empowering Challenging Image Segmentation Task with Freehand Sketches(https://arxiv.org/abs/2501.19329)
• Keywords: segmentation
• Abstract: Sketches, with their expressive potential, allow humans to convey the essence of an object through even a rough contour. For the first time, we harness this expressive potential to improve segmentation performance in challenging tasks like camouflaged object detection (COD). Our approach introduces an innovative sketch-guided interactive segmentation framework, allowing users to intuitively annotate objects with freehand sketches (drawing a rough contour of the object) instead of the traditional bounding boxes or points used in classic interactive segmentation models like SAM. We demonstrate that sketch input can significantly improve performance in existing iterative segmentation methods, outperforming text or bounding box annotations. Additionally, we introduce key modifications to network architectures and a novel sketch augmentation technique to fully harness the power of sketch input and further boost segmentation accuracy. Remarkably, our model' s output can be directly used to train other neural networks, achieving results comparable to pixel-by-pixel annotations--while reducing annotation time by up to 120 times, which shows great potential in democratizing the annotation process and enabling model training with less reliance on resource-intensive, laborious pixel-level annotations. We also present KOSCamo+, the first freehand sketch dataset for camouflaged object detection. The dataset, code, and the labeling tool will be open sourced.

Title: Consistent Video Colorization via Palette Guidance

• Authors: Han Wang, Yuang Zhang, Yuhong Zhang, Lingxiao Lu, Li Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19331
• Pdf URL: https://arxiv.org/pdf/2501.19331
• Copy Paste: [[2501.19331]] Consistent Video Colorization via Palette Guidance(https://arxiv.org/abs/2501.19331)
• Keywords: diffusion, generative
• Abstract: Colorization is a traditional computer vision task and it plays an important role in many time-consuming tasks, such as old film restoration. Existing methods suffer from unsaturated color and temporally inconsistency. In this paper, we propose a novel pipeline to overcome the challenges. We regard the colorization task as a generative task and introduce Stable Video Diffusion (SVD) as our base model. We design a palette-based color guider to assist the model in generating vivid and consistent colors. The color context introduced by the palette not only provides guidance for color generation, but also enhances the stability of the generated colors through a unified color context across multiple sequences. Experiments demonstrate that the proposed method can provide vivid and stable colors for videos, surpassing previous methods.

Title: Homogeneity Bias as Differential Sampling Uncertainty in Language Models

• Authors: Messi H.J. Lee, Soyeon Jeon
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.19337
• Pdf URL: https://arxiv.org/pdf/2501.19337
• Copy Paste: [[2501.19337]] Homogeneity Bias as Differential Sampling Uncertainty in Language Models(https://arxiv.org/abs/2501.19337)
• Keywords: large language model
• Abstract: Prior research show that Large Language Models (LLMs) and Vision-Language Models (VLMs) represent marginalized groups more homogeneously than dominant groups. However, the mechanisms underlying this homogeneity bias remain relatively unexplored. We propose that this bias emerges from systematic differences in the probability distributions from which tokens are sampled at inference-time. Analyzing three measures of uncertainty in token sampling distributions-entropy, perplexity, and probability of differentiation-we find that in some models, specifically GPT-4 Turbo and Llama-3.2, tokens are sampled more deterministically when generating texts about marginalized groups (i.e., Black Americans and women) compared to their dominant group counterparts (i.e., White Americans and men). While these findings may help explain homogeneity bias in certain models, the patterns did not replicate across all VLMs tested, suggesting multiple mechanisms may contribute to homogeneity bias in AI.

Title: The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

• Authors: Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, Dacheng Tao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19358
• Pdf URL: https://arxiv.org/pdf/2501.19358
• Copy Paste: [[2501.19358]] The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking(https://arxiv.org/abs/2501.19358)
• Keywords: large language model
• Abstract: This work identifies the Energy Loss Phenomenon in Reinforcement Learning from Human Feedback (RLHF) and its connection to reward hacking. Specifically, energy loss in the final layer of a Large Language Model (LLM) gradually increases during the RL process, with an excessive increase in energy loss characterizing reward hacking. Beyond empirical analysis, we further provide a theoretical foundation by proving that, under mild conditions, the increased energy loss reduces the upper bound of contextual relevance in LLMs, which is a critical aspect of reward hacking as the reduced contextual relevance typically indicates overfitting to reward model-favored patterns in RL. To address this issue, we propose an Energy loss-aware PPO algorithm (EPPO) which penalizes the increase in energy loss in the LLM's final layer during reward calculation to prevent excessive energy loss, thereby mitigating reward hacking. We theoretically show that EPPO can be conceptually interpreted as an entropy-regularized RL algorithm, which provides deeper insights into its effectiveness. Extensive experiments across various LLMs and tasks demonstrate the commonality of the energy loss phenomenon, as well as the effectiveness of \texttt{EPPO} in mitigating reward hacking and improving RLHF performance.

Title: CoSTI: Consistency Models for (a faster) Spatio-Temporal Imputation

• Authors: Javier Solís-García, Belén Vega-Márquez, Juan A. Nepomuceno, Isabel A. Nepomuceno-Chamorro
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19364
• Pdf URL: https://arxiv.org/pdf/2501.19364
• Copy Paste: [[2501.19364]] CoSTI: Consistency Models for (a faster) Spatio-Temporal Imputation(https://arxiv.org/abs/2501.19364)
• Keywords: diffusion, generative
• Abstract: Multivariate Time Series Imputation (MTSI) is crucial for many applications, such as healthcare monitoring and traffic management, where incomplete data can compromise decision-making. Existing state-of-the-art methods, like Denoising Diffusion Probabilistic Models (DDPMs), achieve high imputation accuracy; however, they suffer from significant computational costs and are notably time-consuming due to their iterative nature. In this work, we propose CoSTI, an innovative adaptation of Consistency Models (CMs) for the MTSI domain. CoSTI employs Consistency Training to achieve comparable imputation quality to DDPMs while drastically reducing inference times, making it more suitable for real-time applications. We evaluate CoSTI across multiple datasets and missing data scenarios, demonstrating up to a 98% reduction in imputation time with performance on par with diffusion-based models. This work bridges the gap between efficiency and accuracy in generative imputation tasks, providing a scalable solution for handling missing data in critical spatio-temporal systems.

Title: LiDAR Loop Closure Detection using Semantic Graphs with Graph Attention Networks

• Authors: Liudi Yang, Ruben Mascaro, Ignacio Alzugaray, Sai Manoj Prakhya, Marco Karrer, Ziyuan Liu, Margarita Chli
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.19382
• Pdf URL: https://arxiv.org/pdf/2501.19382
• Copy Paste: [[2501.19382]] LiDAR Loop Closure Detection using Semantic Graphs with Graph Attention Networks(https://arxiv.org/abs/2501.19382)
• Keywords: robust
• Abstract: In this paper, we propose a novel loop closure detection algorithm that uses graph attention neural networks to encode semantic graphs to perform place recognition and then use semantic registration to estimate the 6 DoF relative pose constraint. Our place recognition algorithm has two key modules, namely, a semantic graph encoder module and a graph comparison module. The semantic graph encoder employs graph attention networks to efficiently encode spatial, semantic and geometric information from the semantic graph of the input point cloud. We then use self-attention mechanism in both node-embedding and graph-embedding steps to create distinctive graph vectors. The graph vectors of the current scan and a keyframe scan are then compared in the graph comparison module to identify a possible loop closure. Specifically, employing the difference of the two graph vectors showed a significant improvement in performance, as shown in ablation studies. Lastly, we implemented a semantic registration algorithm that takes in loop closure candidate scans and estimates the relative 6 DoF pose constraint for the LiDAR SLAM system. Extensive evaluation on public datasets shows that our model is more accurate and robust, achieving 13% improvement in maximum F1 score on the SemanticKITTI dataset, when compared to the baseline semantic graph algorithm. For the benefit of the community, we open-source the complete implementation of our proposed algorithm and custom implementation of semantic registration at this https URL

Title: Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large Language Models

• Authors: Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, Christopher G. Brinton
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19389
• Pdf URL: https://arxiv.org/pdf/2501.19389
• Copy Paste: [[2501.19389]] Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large Language Models(https://arxiv.org/abs/2501.19389)
• Keywords: federate, large language model
• Abstract: Fine-tuning large language models (LLMs) on devices is attracting increasing interest. Recent works have fused low-rank adaptation (LoRA) techniques with federated fine-tuning to mitigate challenges associated with device model sizes and data scarcity. Still, the heterogeneity of computational resources remains a critical bottleneck: while higher-rank modules generally enhance performance, varying device capabilities constrain LoRA's feasible rank range. Existing approaches attempting to resolve this issue either lack analytical justification or impose additional computational overhead, leaving a wide gap for an efficient and theoretically-grounded solution. To address these challenges, we propose federated sketching LoRA (FSLoRA), which leverages a sketching mechanism to enable devices to selectively update submatrices of global LoRA modules maintained by the server. By adjusting the sketching ratios, which determine the ranks of the submatrices on the devices, FSLoRA flexibly adapts to device-specific communication and computational constraints. We provide a rigorous convergence analysis of FSLoRA that characterizes how the sketching ratios affect the convergence rate. Through comprehensive experiments on multiple datasets and LLM models, we demonstrate FSLoRA's superior performance compared to various baselines.

Title: Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

• Authors: Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, Denis Kuznedelev, Denis Mazur, Nikita Surkov, Ivan Ermakov, Dan Alistarh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19392
• Pdf URL: https://arxiv.org/pdf/2501.19392
• Copy Paste: [[2501.19392]] Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models(https://arxiv.org/abs/2501.19392)
• Keywords: large language model
• Abstract: Efficient real-world deployments of large language models (LLMs) rely on Key-Value (KV) caching for processing and generating long outputs, reducing the need for repetitive computation. For large contexts, Key-Value caches can take up tens of gigabytes of device memory, as they store vector representations for each token and layer. Recent work has shown that the cached vectors can be compressed through quantization, pruning or merging, but these techniques often compromise quality towards higher compression rates. In this work, we aim to improve Key & Value compression by exploiting two observations: 1) the inherent dependencies between keys and values across different layers, and 2) high-compression mechanisms for internal network states. We propose AQUA-KV, an adaptive quantization for Key-Value caches that relies on compact adapters to exploit existing dependencies between Keys and Values, and aims to "optimally" compress the information that cannot be predicted. AQUA-KV significantly improves compression rates, while maintaining high accuracy on state-of-the-art LLM families. On Llama 3.2 LLMs, we achieve near-lossless inference at 2-2.5 bits per value with under $1\%$ relative error in perplexity and LongBench scores. AQUA-KV is one-shot, simple, and efficient: it can be calibrated on a single GPU within 1-6 hours, even for 70B models.

Title: Scalable-Softmax Is Superior for Attention

• Authors: Ken M. Nakanishi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19399
• Pdf URL: https://arxiv.org/pdf/2501.19399
• Copy Paste: [[2501.19399]] Scalable-Softmax Is Superior for Attention(https://arxiv.org/abs/2501.19399)
• Keywords: transformer
• Abstract: The maximum element of the vector output by the Softmax function approaches zero as the input vector size increases. Transformer-based language models rely on Softmax to compute attention scores, causing the attention distribution to flatten as the context size grows. This reduces the model's ability to prioritize key information effectively and potentially limits its length generalization. To address this problem, we propose Scalable-Softmax (SSMax), which replaces Softmax in scenarios where the input vector size varies. SSMax can be seamlessly integrated into existing Transformer-based architectures. Experimental results in language modeling show that models using SSMax not only achieve faster loss reduction during pretraining but also significantly improve performance in long contexts and key information retrieval. Furthermore, an analysis of attention scores reveals that SSMax enables the model to focus attention on key information even in long contexts. Additionally, although models that use SSMax from the beginning of pretraining achieve better length generalization, those that have already started pretraining can still gain some of this ability by replacing Softmax in the attention layers with SSMax, either during or after pretraining.

Title: Vintix: Action Model via In-Context Reinforcement Learning

• Authors: Andrey Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Ilya Zisman, Denis Tarasov, Alexander Nikulin, Vladislav Kurenkov
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.19400
• Pdf URL: https://arxiv.org/pdf/2501.19400
• Copy Paste: [[2501.19400]] Vintix: Action Model via In-Context Reinforcement Learning(https://arxiv.org/abs/2501.19400)
• Keywords: large language model
• Abstract: In-Context Reinforcement Learning (ICRL) represents a promising paradigm for developing generalist agents that learn at inference time through trial-and-error interactions, analogous to how large language models adapt contextually, but with a focus on reward maximization. However, the scalability of ICRL beyond toy tasks and single-domain settings remains an open challenge. In this work, we present the first steps toward scaling ICRL by introducing a fixed, cross-domain model capable of learning behaviors through in-context reinforcement learning. Our results demonstrate that Algorithm Distillation, a framework designed to facilitate ICRL, offers a compelling and competitive alternative to expert distillation to construct versatile action models. These findings highlight the potential of ICRL as a scalable approach for generalist decision-making systems. Code to be released at this https URL

Title: Redefining Machine Unlearning: A Conformal Prediction-Motivated Approach

• Authors: Yingdan Shi, Ren Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.19403
• Pdf URL: https://arxiv.org/pdf/2501.19403
• Copy Paste: [[2501.19403]] Redefining Machine Unlearning: A Conformal Prediction-Motivated Approach(https://arxiv.org/abs/2501.19403)
• Keywords: attack, membership infer
• Abstract: Machine unlearning seeks to systematically remove specified data from a trained model, effectively achieving a state as though the data had never been encountered during training. While metrics such as Unlearning Accuracy (UA) and Membership Inference Attack (MIA) provide a baseline for assessing unlearning performance, they fall short of evaluating the completeness and reliability of forgetting. This is because the ground truth labels remain potential candidates within the scope of uncertainty quantification, leaving gaps in the evaluation of true forgetting. In this paper, we identify critical limitations in existing unlearning metrics and propose enhanced evaluation metrics inspired by conformal prediction. Our metrics can effectively capture the extent to which ground truth labels are excluded from the prediction set. Furthermore, we observe that many existing machine unlearning methods do not achieve satisfactory forgetting performance when evaluated with our new metrics. To address this, we propose an unlearning framework that integrates conformal prediction insights into Carlini & Wagner adversarial attack loss. Extensive experiments on the image classification task demonstrate that our enhanced metrics offer deeper insights into unlearning effectiveness, and that our unlearning framework significantly improves the forgetting quality of unlearning methods.

Title: Low-Rank Adapting Models for Sparse Autoencoders

• Authors: Matthew Chen, Joshua Engels, Max Tegmark
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.19406
• Pdf URL: https://arxiv.org/pdf/2501.19406
• Copy Paste: [[2501.19406]] Low-Rank Adapting Models for Sparse Autoencoders(https://arxiv.org/abs/2501.19406)
• Keywords: interpretability
• Abstract: Sparse autoencoders (SAEs) decompose language model representations into a sparse set of linear latent vectors. Recent works have improved SAEs using language model gradients, but these techniques require many expensive backward passes during training and still cause a significant increase in cross entropy loss when SAE reconstructions are inserted into the model. In this work, we improve on these limitations by taking a fundamentally different approach: we use low-rank adaptation (LoRA) to finetune the language model itself around a previously trained SAE. We analyze our method across SAE sparsity, SAE width, language model size, LoRA rank, and model layer on the Gemma Scope family of SAEs. In these settings, our method reduces the cross entropy loss gap by 30% to 55% when SAEs are inserted during the forward pass. We also find that compared to end-to-end (e2e) SAEs, our approach achieves the same downstream cross entropy loss 3$\times$ to 20$\times$ faster on Gemma-2-2B and 2$\times$ to 10$\times$ faster on Llama-3.2-1B. We further show that our technique improves downstream metrics and can adapt multiple SAEs at once. Our results demonstrate that improving model interpretability is not limited to post-hoc SAE training; Pareto improvements can also be achieved by directly optimizing the model itself.