2025-01-24

Title: Dagger Behind Smile: Fool LLMs with a Happy Ending Story

• Authors: Xurui Song, Zhixin Xie, Shuo Huai, Jiayi Kong, Jun Luo
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13115
• Pdf URL: https://arxiv.org/pdf/2501.13115
• Copy Paste: [[2501.13115]] Dagger Behind Smile: Fool LLMs with a Happy Ending Story(https://arxiv.org/abs/2501.13115)
• Keywords: attack, large language model
• Abstract: The wide adoption of Large Language Models (LLMs) has attracted significant attention from \textit{jailbreak} attacks, where adversarial prompts crafted through optimization or manual design exploit LLMs to generate malicious content. However, optimization-based attacks have limited efficiency and transferability, while manual designs are either easily detectable or demand intricate interactions with LLMs. In this paper, we first point out a novel perspective for jailbreak attacks: LLMs are more responsive to \textit{positive} prompts. Based on this, we deploy Happy Ending Attack (HEA) to wrap up a malicious request in a scenario template involving a positive prompt formed mainly via a \textit{happy ending}, it thus fools LLMs into jailbreaking either immediately or at a follow-up malicious request. This has made HEA both efficient and effective, as it requires only up to two steps to fully jailbreak LLMs. Extensive experiments show that our HEA can successfully jailbreak on state-of-the-art LLMs, including GPT-4o, Llama3-70b, Gemini-pro, and achieves 88.79\% Attack Success Rate on average. We also provide potential quantitative explanations for the success of HEA.

Title: MyGO Multiplex CoT: A Method for Self-Reflection in Large Language Models via Double Chain of Thought Thinking

• Authors: Shihao Ji, Zihui Song, Fucheng Zhong, Jisen Jia, Zhaobo Wu, Zheyi Cao, Tianhao Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13117
• Pdf URL: https://arxiv.org/pdf/2501.13117
• Copy Paste: [[2501.13117]] MyGO Multiplex CoT: A Method for Self-Reflection in Large Language Models via Double Chain of Thought Thinking(https://arxiv.org/abs/2501.13117)
• Keywords: robust, large language model
• Abstract: Recent advancements in large language models (LLMs) have demonstrated their impressive abilities in various reasoning and decision-making tasks. However, the quality and coherence of the reasoning process can still benefit from enhanced introspection and self-reflection. In this paper, we introduce Multiplex CoT (Chain of Thought), a method that enables LLMs to simulate a form of self-review while reasoning, by initiating double Chain of Thought (CoT) thinking. Multiplex CoT leverages the power of iterative reasoning, where the model generates an initial chain of thought and subsequently critiques and refines this reasoning with a second round of thought generation. This recursive approach allows for more coherent, logical, and robust answers, improving the overall decision-making process. We demonstrate how this method can be effectively implemented using simple prompt engineering in existing LLM architectures, achieving an effect similar to that of the Learning-Refinement Model (LRM) without the need for additional training. Additionally, we present a practical guide for implementing the method in Google Colab, enabling easy integration into real-world applications.

Title: Multilinguality in LLM-Designed Reward Functions for Restless Bandits: Effects on Task Performance and Fairness

• Authors: Ambreesh Parthasarathy, Chandrasekar Subramanian, Ganesh Senrayan, Shreyash Adappanavar, Aparna Taneja, Balaraman Ravindran, Milind Tambe
• Subjects: cs.CL, cs.AI, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.13120
• Pdf URL: https://arxiv.org/pdf/2501.13120
• Copy Paste: [[2501.13120]] Multilinguality in LLM-Designed Reward Functions for Restless Bandits: Effects on Task Performance and Fairness(https://arxiv.org/abs/2501.13120)
• Keywords: robust, fair, large language model
• Abstract: Restless Multi-Armed Bandits (RMABs) have been successfully applied to resource allocation problems in a variety of settings, including public health. With the rapid development of powerful large language models (LLMs), they are increasingly used to design reward functions to better match human preferences. Recent work has shown that LLMs can be used to tailor automated allocation decisions to community needs using language prompts. However, this has been studied primarily for English prompts and with a focus on task performance only. This can be an issue since grassroots workers, especially in developing countries like India, prefer to work in local languages, some of which are low-resource. Further, given the nature of the problem, biases along population groups unintended by the user are also undesirable. In this work, we study the effects on both task performance and fairness when the DLM algorithm, a recent work on using LLMs to design reward functions for RMABs, is prompted with non-English language commands. Specifically, we run the model on a synthetic environment for various prompts translated into multiple languages. The prompts themselves vary in complexity. Our results show that the LLM-proposed reward functions are significantly better when prompted in English compared to other languages. We also find that the exact phrasing of the prompt impacts task performance. Further, as prompt complexity increases, performance worsens for all languages; however, it is more robust with English prompts than with lower-resource languages. On the fairness side, we find that low-resource languages and more complex prompts are both highly likely to create unfairness along unintended dimensions.

Title: Episodic Memories Generation and Evaluation Benchmark for Large Language Models

• Authors: Alexis Huet, Zied Ben Houidi, Dario Rossi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13121
• Pdf URL: https://arxiv.org/pdf/2501.13121
• Copy Paste: [[2501.13121]] Episodic Memories Generation and Evaluation Benchmark for Large Language Models(https://arxiv.org/abs/2501.13121)
• Keywords: robust, large language model
• Abstract: Episodic memory -- the ability to recall specific events grounded in time and space -- is a cornerstone of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack a robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confabulations. To address this challenge, we introduce a comprehensive framework to model and evaluate LLM episodic memory capabilities. Drawing inspiration from cognitive science, we develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from contamination, and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-art models, including GPT-4 and Claude variants, Llama 3.1, and o1-mini, reveals that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships -- even in contexts as short as 10k-100k tokens.

Title: Zero-Shot Verification-guided Chain of Thoughts

• Authors: Jishnu Ray Chowdhury, Cornelia Caragea
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13122
• Pdf URL: https://arxiv.org/pdf/2501.13122
• Copy Paste: [[2501.13122]] Zero-Shot Verification-guided Chain of Thoughts(https://arxiv.org/abs/2501.13122)
• Keywords: large language model
• Abstract: Previous works have demonstrated the effectiveness of Chain-of-Thought (COT) prompts and verifiers in guiding Large Language Models (LLMs) through the space of reasoning. However, most such studies either use a fine-tuned verifier or rely on manually handcrafted few-shot examples. In contrast, in this paper, we focus on LLM-based self-verification of self-generated reasoning steps via COT prompts in a completely zero-shot regime. To explore this setting, we design a new zero-shot prompt, which we call COT STEP, to aid zero-shot decomposition of reasoning steps and design two new zero-shot prompts for LLM-based verifiers. We evaluate the verifiers' ability to classify the correctness of reasoning chains and explore different ways to use verifier scores in guiding reasoning for various mathematical and commonsense reasoning tasks with different LLMs.

Title: Debate Helps Weak-to-Strong Generalization

• Authors: Hao Lang, Fei Huang, Yongbin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13124
• Pdf URL: https://arxiv.org/pdf/2501.13124
• Copy Paste: [[2501.13124]] Debate Helps Weak-to-Strong Generalization(https://arxiv.org/abs/2501.13124)
• Keywords: robust
• Abstract: Common methods for aligning already-capable models with desired behavior rely on the ability of humans to provide supervision. However, future superhuman models will surpass the capability of humans. Therefore, humans will only be able to weakly supervise superhuman models. This expected deficiency of human evaluation would weaken the safety of future AI systems. Scalable oversight and weak-to-strong generalization are two complementary approaches to tackle this issue. In this paper, we attempt to combine the strengths of these two approaches to further improve alignment. Specifically, we investigate ways of improving human supervision with a strong pretrained model and then supervise the strong model with enhanced weak human supervision. To make iterative empirical progress, we consider an analogy: can we use a strong model to improve weak model supervision and then use it to supervise the strong model? We empirically test it by finetuning a small weak model on ground truth labels with the additional help from a large strong model, and then finetuning the strong model on labels generated by the weak model. We find that debate can assist a weak model in extracting trustworthy information from an untrustworthy strong model, which provides leverage as context on samples when training a weak model. We also show that an ensemble of weak models helps exploit long arguments generated by strong model debaters and obtain a more robust supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP benchmarks show that the combination approach leads to better alignment, which indicates that debate has the potential to help weak-to-strong generalization.

Title: Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data

• Authors: Xuemiao Zhang, Liangyu Xu, Feiyu Duan, Yongwei Zhou, Sirui Wang, Jingang Wang, Xunliang Cai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13126
• Pdf URL: https://arxiv.org/pdf/2501.13126
• Copy Paste: [[2501.13126]] Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data(https://arxiv.org/abs/2501.13126)
• Keywords: large language model
• Abstract: Current large language models (LLMs) generally utilize a consistent data distribution throughout the entire pretraining process. However, as the model's ability improves, it intuitively should be pretrained with differentiated data. To achieve it, we propose the Perplexity Difference based Preference Curriculum learning (PDPC) framework, which always perceives and uses the data preferred by LLMs to train and boost them. Firstly, we introduce the PD metric to measure the difference in how well strong and weak models fit the samples. Samples with high PD are more challenging for weak models to learn and are more suitable to be arranged in the later stage of pretraining. Secondly, we propose the PD preference function to approximate the model and predict the data preference of the LLM at any time, so as to complete the arrangement of the entire data offline and ensure continuous training without interruption. Experimental results on 1.3B and 3B models demonstrate that our PDPC significantly surpasses baselines. Notably, the 3B model achieved more substantial gains, with an increased average accuracy of over 4.1% across various benchmarks.

Title: Graph Representation Learning with Diffusion Generative Models

• Authors: Daniel Wesego
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13133
• Pdf URL: https://arxiv.org/pdf/2501.13133
• Copy Paste: [[2501.13133]] Graph Representation Learning with Diffusion Generative Models(https://arxiv.org/abs/2501.13133)
• Keywords: diffusion, generative
• Abstract: Diffusion models have established themselves as state-of-the-art generative models across various data modalities, including images and videos, due to their ability to accurately approximate complex data distributions. Unlike traditional generative approaches such as VAEs and GANs, diffusion models employ a progressive denoising process that transforms noise into meaningful data over multiple iterative steps. This gradual approach enhances their expressiveness and generation quality. Not only that, diffusion models have also been shown to extract meaningful representations from data while learning to generate samples. Despite their success, the application of diffusion models to graph-structured data remains relatively unexplored, primarily due to the discrete nature of graphs, which necessitates discrete diffusion processes distinct from the continuous methods used in other domains. In this work, we leverage the representational capabilities of diffusion models to learn meaningful embeddings for graph data. By training a discrete diffusion model within an autoencoder framework, we enable both effective autoencoding and representation learning tailored to the unique characteristics of graph-structured data. We only need the encoder at the end to extract representations. Our approach demonstrates the potential of discrete diffusion models to be used for graph representation learning.

Title: MONA: Moving Object Detection from Videos Shot by Dynamic Camera

• Authors: Boxun Hu, Mingze Xia, Ding Zhao, Guanlin Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13183
• Pdf URL: https://arxiv.org/pdf/2501.13183
• Copy Paste: [[2501.13183]] MONA: Moving Object Detection from Videos Shot by Dynamic Camera(https://arxiv.org/abs/2501.13183)
• Keywords: robust, extraction, segmentation
• Abstract: Dynamic urban environments, characterized by moving cameras and objects, pose significant challenges for camera trajectory estimation by complicating the distinction between camera-induced and object motion. We introduce MONA, a novel framework designed for robust moving object detection and segmentation from videos shot by dynamic cameras. MONA comprises two key modules: Dynamic Points Extraction, which leverages optical flow and tracking any point to identify dynamic points, and Moving Object Segmentation, which employs adaptive bounding box filtering, and the Segment Anything for precise moving object segmentation. We validate MONA by integrating with the camera trajectory estimation method LEAP-VO, and it achieves state-of-the-art results on the MPI Sintel dataset comparing to existing methods. These results demonstrate MONA's effectiveness for moving object detection and its potential in many other applications in the urban planning field.

Title: SRMT: Shared Memory for Multi-agent Lifelong Pathfinding

• Authors: Alsu Sagirova, Yuri Kuratov, Mikhail Burtsev
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.13200
• Pdf URL: https://arxiv.org/pdf/2501.13200
• Copy Paste: [[2501.13200]] SRMT: Shared Memory for Multi-agent Lifelong Pathfinding(https://arxiv.org/abs/2501.13200)
• Keywords: transformer
• Abstract: Multi-agent reinforcement learning (MARL) demonstrates significant progress in solving cooperative and competitive multi-agent problems in various environments. One of the principal challenges in MARL is the need for explicit prediction of the agents' behavior to achieve cooperation. To resolve this issue, we propose the Shared Recurrent Memory Transformer (SRMT) which extends memory transformers to multi-agent settings by pooling and globally broadcasting individual working memories, enabling agents to exchange information implicitly and coordinate their actions. We evaluate SRMT on the Partially Observable Multi-Agent Pathfinding problem in a toy Bottleneck navigation task that requires agents to pass through a narrow corridor and on a POGEMA benchmark set of tasks. In the Bottleneck task, SRMT consistently outperforms a variety of reinforcement learning baselines, especially under sparse rewards, and generalizes effectively to longer corridors than those seen during training. On POGEMA maps, including Mazes, Random, and MovingAI, SRMT is competitive with recent MARL, hybrid, and planning-based algorithms. These results suggest that incorporating shared recurrent memory into the transformer-based architectures can enhance coordination in decentralized multi-agent systems. The source code for training and evaluation is available on GitHub: this https URL.

Title: Distributed Intrusion Detection in Dynamic Networks of UAVs using Few-Shot Federated Learning

• Authors: Ozlem Ceviz, Sevil Sen, Pinar Sadioglu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13213
• Pdf URL: https://arxiv.org/pdf/2501.13213
• Copy Paste: [[2501.13213]] Distributed Intrusion Detection in Dynamic Networks of UAVs using Few-Shot Federated Learning(https://arxiv.org/abs/2501.13213)
• Keywords: security, privacy, attack, federate
• Abstract: Flying Ad Hoc Networks (FANETs), which primarily interconnect Unmanned Aerial Vehicles (UAVs), present distinctive security challenges due to their distributed and dynamic characteristics, necessitating tailored security solutions. Intrusion detection in FANETs is particularly challenging due to communication costs, and privacy concerns. While Federated Learning (FL) holds promise for intrusion detection in FANETs with its cooperative and decentralized model training, it also faces drawbacks such as large data requirements, power consumption, and time constraints. Moreover, the high speeds of nodes in dynamic networks like FANETs may disrupt communication among Intrusion Detection Systems (IDS). In response, our study explores the use of few-shot learning (FSL) to effectively reduce the data required for intrusion detection in FANETs. The proposed approach called Few-shot Federated Learning-based IDS (FSFL-IDS) merges FL and FSL to tackle intrusion detection challenges such as privacy, power constraints, communication costs, and lossy links, demonstrating its effectiveness in identifying routing attacks in dynamic this http URL approach reduces both the local models and the global model's training time and sample size, offering insights into reduced computation and communication costs and extended battery life. Furthermore, by employing FSL, which requires less data for training, IDS could be less affected by lossy links in FANETs.

Title: Enhancing Multi-Attribute Fairness in Healthcare Predictive Modeling

• Authors: Xiaoyang Wang, Christopher C. Yang
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.13219
• Pdf URL: https://arxiv.org/pdf/2501.13219
• Copy Paste: [[2501.13219]] Enhancing Multi-Attribute Fairness in Healthcare Predictive Modeling(https://arxiv.org/abs/2501.13219)
• Keywords: fair
• Abstract: Artificial intelligence (AI) systems in healthcare have demonstrated remarkable potential to improve patient outcomes. However, if not designed with fairness in mind, they also carry the risks of perpetuating or exacerbating existing health disparities. Although numerous fairness-enhancing techniques have been proposed, most focus on a single sensitive attribute and neglect the broader impact that optimizing fairness for one attribute may have on the fairness of other sensitive attributes. In this work, we introduce a novel approach to multi-attribute fairness optimization in healthcare AI, tackling fairness concerns across multiple demographic attributes concurrently. Our method follows a two-phase approach: initially optimizing for predictive performance, followed by fine-tuning to achieve fairness across multiple sensitive attributes. We develop our proposed method using two strategies, sequential and simultaneous. Our results show a significant reduction in Equalized Odds Disparity (EOD) for multiple attributes, while maintaining high predictive accuracy. Notably, we demonstrate that single-attribute fairness methods can inadvertently increase disparities in non-targeted attributes whereas simultaneous multi-attribute optimization achieves more balanced fairness improvements across all attributes. These findings highlight the importance of comprehensive fairness strategies in healthcare AI and offer promising directions for future research in this critical area.

Title: Scaling for Fairness? Analyzing Model Size, Data Composition, and Multilinguality in Vision-Language Bias

• Authors: Zahraa Al Sahili, Ioannis Patras, Matthew Purver
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13223
• Pdf URL: https://arxiv.org/pdf/2501.13223
• Copy Paste: [[2501.13223]] Scaling for Fairness? Analyzing Model Size, Data Composition, and Multilinguality in Vision-Language Bias(https://arxiv.org/abs/2501.13223)
• Keywords: fair
• Abstract: As large-scale vision-language models (VLMs) become increasingly central to modern AI applications, understanding and mitigating social biases in these systems has never been more this http URL investigate how dataset composition, model size, and multilingual training affect gender and racial bias in a popular VLM, CLIP, and its open-source variants. In particular, we systematically evaluate models trained on varying dataset scales and architectures, as well as multilingual versions encompassing English along with Persian, Turkish, and Finnish, languages with minimal gender marking. To assess social perception bias, we measure the zero-shot performance on face images featuring socially charged terms rooted in the psychological constructs of communion and agency, and demographic labeling bias using both the FairFace and PATA datasets. Our findings reveal three key insights. First, while larger training datasets can mitigate some biases, they may also introduce or amplify others when the data composition is imbalanced. Second, although increasing model size generally improves performance, it does not consistently reduce bias and can, in certain cases, exacerbate it. Finally, while multilingual training broadens linguistic coverage, it does not inherently neutralize bias and can transfer or intensify inequities across languages. Taken together, these results highlight the necessity of inclusive, carefully curated training data to foster fairness rather than relying solely on model scaling or language expansion. We provide a systematic evaluation of vision language bias across diverse demographics, underscoring the urgent need for intentional bias mitigation strategies in next generation AI systems.

Title: Joint Task Offloading and User Scheduling in 5G MEC under Jamming Attacks

• Authors: Mohammadreza Amini, Burak Kantarci, Claude D'Amours, Melike Erol-Kantarci
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.13227
• Pdf URL: https://arxiv.org/pdf/2501.13227
• Copy Paste: [[2501.13227]] Joint Task Offloading and User Scheduling in 5G MEC under Jamming Attacks(https://arxiv.org/abs/2501.13227)
• Keywords: security, attack
• Abstract: In this paper, we propose a novel joint task offloading and user scheduling (JTO-US) framework for 5G mobile edge computing (MEC) systems under security threats from jamming attacks. The goal is to minimize the delay and the ratio of dropped tasks, taking into account both communication and computation delays. The system model includes a 5G network equipped with MEC servers and an adversarial on-off jammer that disrupts communication. The proposed framework optimally schedules tasks and users to minimize the impact of jamming while ensuring that high-priority tasks are processed efficiently. Genetic algorithm (GA) is used to solve the optimization problem, and the results are compared with benchmark methods such as GA without considering jamming effect, Shortest Job First (SJF), and Shortest Deadline First (SDF). The simulation results demonstrate that the proposed JTO-US framework achieves the lowest drop ratio and effectively manages priority tasks, outperforming existing methods. Particularly, when the jamming probability is 0.8, the proposed framework mitigates the jammer's impact by reducing the drop ratio to 63%, compared to 89% achieved by the next best method.

Title: Active RIS-Assisted URLLC NOMA-Based 5G Network with FBL under Jamming Attacks

• Authors: Ghazal Asemian, Mohammadreza Amini, Burak Kantarci
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.13231
• Pdf URL: https://arxiv.org/pdf/2501.13231
• Copy Paste: [[2501.13231]] Active RIS-Assisted URLLC NOMA-Based 5G Network with FBL under Jamming Attacks(https://arxiv.org/abs/2501.13231)
• Keywords: attack
• Abstract: In this paper, we tackle the challenge of jamming attacks in Ultra-Reliable Low Latency Communication (URLLC) within Non-Orthogonal Multiple Access (NOMA)-based 5G networks under Finite Blocklength (FBL) conditions. We introduce an innovative approach that employs Reconfigurable Intelligent Surfaces (RIS) with active elements to enhance energy efficiency while ensuring reliability and meeting latency requirements. Our approach incorporates the traffic model, making it practical for real-world scenarios with dynamic traffic loads. We thoroughly analyze the impact of blocklength and packet arrival rate on network performance metrics and investigate the optimal amplitude value and number of RIS elements. Our results indicate that increasing the number of RIS elements from 4 to 400 can improve signal-to-jamming-plus-noise ratio (SJNR) by 13.64\%. Additionally, optimizing blocklength and packet arrival rate can achieve a 31.68% improvement in energy efficiency and reduced latency. These findings underscore the importance of optimized settings for effective jamming mitigation.

Title: State Combinatorial Generalization In Decision Making With Conditional Diffusion Models

• Authors: Xintong Duan, Yutong He, Fahim Tajwar, Wen-Tse Chen, Ruslan Salakhutdinov, Jeff Schneider
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13241
• Pdf URL: https://arxiv.org/pdf/2501.13241
• Copy Paste: [[2501.13241]] State Combinatorial Generalization In Decision Making With Conditional Diffusion Models(https://arxiv.org/abs/2501.13241)
• Keywords: diffusion
• Abstract: Many real-world decision-making problems are combinatorial in nature, where states (e.g., surrounding traffic of a self-driving car) can be seen as a combination of basic elements (e.g., pedestrians, trees, and other cars). Due to combinatorial complexity, observing all combinations of basic elements in the training set is infeasible, which leads to an essential yet understudied problem of zero-shot generalization to states that are unseen combinations of previously seen elements. In this work, we first formalize this problem and then demonstrate how existing value-based reinforcement learning (RL) algorithms struggle due to unreliable value predictions in unseen states. We argue that this problem cannot be addressed with exploration alone, but requires more expressive and generalizable models. We demonstrate that behavior cloning with a conditioned diffusion model trained on expert trajectory generalizes better to states formed by new combinations of seen elements than traditional RL methods. Through experiments in maze, driving, and multiagent environments, we show that conditioned diffusion models outperform traditional RL techniques and highlight the broad applicability of our problem formulation.

Title: Multimodal AI on Wound Images and Clinical Notes for Home Patient Referral

• Authors: Reza Saadati Fard, Emmanuel Agu, Palawat Busaranuvong, Deepak Kumar, Shefalika Gautam, Bengisu Tulu, Diane Strong
• Subjects: cs.LG, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.13247
• Pdf URL: https://arxiv.org/pdf/2501.13247
• Copy Paste: [[2501.13247]] Multimodal AI on Wound Images and Clinical Notes for Home Patient Referral(https://arxiv.org/abs/2501.13247)
• Keywords: interpretability, transformer
• Abstract: Chronic wounds affect 8.5 million Americans, particularly the elderly and patients with diabetes. These wounds can take up to nine months to heal, making regular care essential to ensure healing and prevent severe outcomes like limb amputations. Many patients receive care at home from visiting nurses with varying levels of wound expertise, leading to inconsistent care. Problematic, non-healing wounds should be referred to wound specialists, but referral decisions in non-clinical settings are often erroneous, delayed, or unnecessary. This paper introduces the Deep Multimodal Wound Assessment Tool (DM-WAT), a machine learning framework designed to assist visiting nurses in deciding whether to refer chronic wound patients. DM-WAT analyzes smartphone-captured wound images and clinical notes from Electronic Health Records (EHRs). It uses DeiT-Base-Distilled, a Vision Transformer (ViT), to extract visual features from images and DeBERTa-base to extract text features from clinical notes. DM-WAT combines visual and text features using an intermediate fusion approach. To address challenges posed by a small and imbalanced dataset, it integrates image and text augmentation with transfer learning to achieve high performance. In evaluations, DM-WAT achieved 77% with std 3% accuracy and a 70% with std 2% F1 score, outperforming prior approaches. Score-CAM and Captum interpretation algorithms provide insights into specific parts of image and text inputs that influence recommendations, enhancing interpretability and trust.

Title: Exploring the Technology Landscape through Topic Modeling, Expert Involvement, and Reinforcement Learning

• Authors: Ali Nazari, Michael Weiss
• Subjects: cs.LG, cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2501.13252
• Pdf URL: https://arxiv.org/pdf/2501.13252
• Copy Paste: [[2501.13252]] Exploring the Technology Landscape through Topic Modeling, Expert Involvement, and Reinforcement Learning(https://arxiv.org/abs/2501.13252)
• Keywords: security, robust
• Abstract: This study presents a method for exploring advancements in a specific technological domain. It combines topic modeling, expert input, and reinforcement learning (RL). The proposed approach has three key steps: (1) generate aspect-based topic models using expert-weighted keywords to emphasize critical aspects, (2) analyze similarities and entropy changes by comparing topic distributions across iterative models, and (3) refine topic selection using reinforcement learning (RL) with a modified reward function that integrates changes in topic divergence and similarity across iterations. The method is tested on quantum communication documents with a focus on advances in cryptography and security protocols. The results show the method's effectiveness and can identify, rank, and track trends that match expert input. The framework provides a robust tool for exploring evolving technological landscapes.

Title: Bypassing Array Canaries via Autonomous Function Call Resolution

• Authors: Nathaniel Oh, Paul Attie, Anas Obeidat
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13256
• Pdf URL: https://arxiv.org/pdf/2501.13256
• Copy Paste: [[2501.13256]] Bypassing Array Canaries via Autonomous Function Call Resolution(https://arxiv.org/abs/2501.13256)
• Keywords: security, large language model
• Abstract: We observed the Array Canary, a novel JavaScript anti-analysis technique currently exploited in-the-wild by the Phishing-as-a-Service framework Darcula. The Array Canary appears to be an advanced form of the array shuffling techniques employed by the Emotet JavaScript downloader. In practice, a series of Array Canaries are set within a string array and if modified will cause the program to endlessly loop. In this paper, we demonstrate how an Array Canary works and discuss Autonomous Function Call Resolution (AFCR), which is a method we created to bypass Array Canaries. We also introduce Arphsy, a proof-of-concept for AFCR designed to guide Large Language Models and security researchers in the deobfuscation of "canaried" JavaScript code. We accomplish this by (i) Finding and extracting all Immediately Invoked Function Expressions from a canaried file, (ii) parsing the file's Abstract Syntax Tree for any function that does not implement imported function calls, (iii) identifying the most reassigned variable and its corresponding function body, (iv) calculating the length of the largest string array and uses it to determine the offset values within the canaried file, (v) aggregating all the previously identified functions into a single file, and (vi) appending driver code into the verified file and using it to deobfuscate the canaried file.

Title: RAG-Reward: Optimizing RAG with Reward Modeling and RLHF

• Authors: Hanning Zhang, Juntong Song, Juno Zhu, Yuanhao Wu, Tong Zhang, Cheng Niu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13264
• Pdf URL: https://arxiv.org/pdf/2501.13264
• Copy Paste: [[2501.13264]] RAG-Reward: Optimizing RAG with Reward Modeling and RLHF(https://arxiv.org/abs/2501.13264)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) enhances Large Language Models (LLMs) with relevant and up-to-date knowledge, improving their ability to answer knowledge-intensive questions. It has been shown to enhance both generation quality and trustworthiness. While numerous works have focused on improving retrieval, generation, and evaluation, the role of reward models in reinforcement learning for optimizing RAG and establishing automated benchmarking pipelines remains underexplored. In this paper, we introduce \textbf{RAG-Reward}, a dataset designed to enable \textit{hallucination-free, comprehensive, reliable, and efficient RAG}. We define four key metrics for assessing generation quality and develop an automated annotation pipeline that leverages multiple LLMs to generate outputs across diverse RAG scenarios. GPT-4o is used to evaluate and construct preference data. Using \textbf{RAG-Reward}, we train reward models and apply reinforcement learning with human feedback (RLHF) to improve LLMs' effectiveness in RAG. Experimental results show that our reward model achieves state-of-the-art performance on a held-out test set, demonstrating both the effectiveness of our approach and the quality of our dataset. Furthermore, the improved generation quality of the trained policy model highlights the feasibility of using RLHF to enhance RAG pipelines.

Title: Threat-based Security Controls to Protect Industrial Control Systems

• Authors: Maryam Karimi, Haritha Srinivasan
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.13268
• Pdf URL: https://arxiv.org/pdf/2501.13268
• Copy Paste: [[2501.13268]] Threat-based Security Controls to Protect Industrial Control Systems(https://arxiv.org/abs/2501.13268)
• Keywords: security, protect
• Abstract: This paper analyzes the reported threats to Industrial Control Systems (ICS)/Operational Technology (OT) and identifies common tactics, techniques, and procedures (TTP) used by threat actors. The paper then uses the MITRE ATT&CK framework to map the common TTPs and provide an understanding of the security controls needed to defend against the reported ICS threats. The paper also includes a review of ICS testbeds and ideas for future research using the identified controls.

Title: Hybrid Two-Stage Reconstruction of Multiscale Subsurface Flow with Physics-informed Residual Connected Neural Operator

• Authors: Peiqi Li, Jie Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13271
• Pdf URL: https://arxiv.org/pdf/2501.13271
• Copy Paste: [[2501.13271]] Hybrid Two-Stage Reconstruction of Multiscale Subsurface Flow with Physics-informed Residual Connected Neural Operator(https://arxiv.org/abs/2501.13271)
• Keywords: transformer
• Abstract: The novel neural networks show great potential in solving partial differential equations. For single-phase flow problems in subsurface porous media with high-contrast coefficients, the key is to develop neural operators with accurate reconstruction capability and strict adherence to physical laws. In this study, we proposed a hybrid two-stage framework that uses multiscale basis functions and physics-guided deep learning to solve the Darcy flow problem in high-contrast fractured porous media. In the first stage, a data-driven model is used to reconstruct the multiscale basis function based on the permeability field to achieve effective dimensionality reduction while preserving the necessary multiscale features. In the second stage, the physics-informed neural network, together with Transformer-based global information extractor is used to reconstruct the pressure field by integrating the physical constraints derived from the Darcy equation, ensuring consistency with the physical laws of the real world. The model was evaluated on datasets with different combinations of permeability and basis functions and performed well in terms of reconstruction accuracy. Specifically, the framework achieves R2 values above 0.9 in terms of basis function fitting and pressure reconstruction, and the residual indicator is on the order of $1\times 10^{-4}$. These results validate the ability of the proposed framework to achieve accurate reconstruction while maintaining physical consistency.

Title: Enhancing Robust Fairness via Confusional Spectral Regularization

• Authors: Gaojie Jin, Sihao Wu, Jiaxu Liu, Tianjin Huang, Ronghui Mu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13273
• Pdf URL: https://arxiv.org/pdf/2501.13273
• Copy Paste: [[2501.13273]] Enhancing Robust Fairness via Confusional Spectral Regularization(https://arxiv.org/abs/2501.13273)
• Keywords: robust, fair
• Abstract: Recent research has highlighted a critical issue known as ``robust fairness", where robust accuracy varies significantly across different classes, undermining the reliability of deep neural networks (DNNs). A common approach to address this has been to dynamically reweight classes during training, giving more weight to those with lower empirical robust performance. However, we find there is a divergence of class-wise robust performance between training set and testing set, which limits the effectiveness of these explicit reweighting methods, indicating the need for a principled alternative. In this work, we derive a robust generalization bound for the worst-class robust error within the PAC-Bayesian framework, accounting for unknown data distributions. Our analysis shows that the worst-class robust error is influenced by two main factors: the spectral norm of the empirical robust confusion matrix and the information embedded in the model and training set. While the latter has been extensively studied, we propose a novel regularization technique targeting the spectral norm of the robust confusion matrix to improve worst-class robust accuracy and enhance robust fairness. We validate our approach through comprehensive experiments on various datasets and models, demonstrating its effectiveness in enhancing robust fairness.

Title: T-Graphormer: Using Transformers for Spatiotemporal Forecasting

• Authors: Hao Yuan Bai, Xue Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13274
• Pdf URL: https://arxiv.org/pdf/2501.13274
• Copy Paste: [[2501.13274]] T-Graphormer: Using Transformers for Spatiotemporal Forecasting(https://arxiv.org/abs/2501.13274)
• Keywords: transformer
• Abstract: Time series data is ubiquitous and appears in all fields of study. In multivariate time series, observations are interconnected both temporally and across components. For instance, in traffic flow analysis, traffic speeds at different intersections exhibit complex spatiotemporal correlations. Modelling this dual structure poses significant challenges. Most existing forecasting methods tackle these challenges by separately learning spatial and temporal dependencies. In this work, we introduce T-Graphormer, a Transformer-based approach designed to model spatiotemporal correlations directly. Extending the Graphormer architecture to incorporate temporal dynamics, our method updates each node representation by selectively attending to all other nodes within a graph sequence. This design enables the model to capture rich spatiotemporal patterns with minimal reliance on predefined spacetime inductive biases. We validate the effectiveness of T-Graphormer on real-world traffic prediction benchmark datasets, achieving up to 10% reductions in both root mean squared error (RMSE) and mean absolute percentage error (MAPE) compared to state-of-the-art methods.

Title: Extraction of Secrets from 40nm CMOS Gate Dielectric Breakdown Antifuses by FIB Passive Voltage Contrast

• Authors: Andrew D. Zonenberg (1), Antony Moor (1), Daniel Slone (1), Lain Agan (1), Mario Cop (1) ((1) IOActive, Seattle, WA)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13276
• Pdf URL: https://arxiv.org/pdf/2501.13276
• Copy Paste: [[2501.13276]] Extraction of Secrets from 40nm CMOS Gate Dielectric Breakdown Antifuses by FIB Passive Voltage Contrast(https://arxiv.org/abs/2501.13276)
• Keywords: security, attack, extraction
• Abstract: CMOS one-time-programmable (OTP) memories based on antifuses are widely used for storing small amounts of data (such as serial numbers, keys, and factory trimming) in integrated circuits due to their low cost, requiring no additional mask steps to fabricate. Device manufacturers and IP vendors have claimed for years that antifuses are a ``high security" memory which is significantly more difficult for an attacker to extract data from than other types of memory, such as Flash or mask ROM - however, as our results show, this is untrue. In this paper, we demonstrate that data bits stored in a widely used antifuse block can be extracted by a semiconductor failure analysis technique known as passive voltage contrast (PVC) using a focused ion beam (FIB). The simple form of the attack demonstrated here recovers the bitwise OR of two physically adjacent memory rows sharing common metal 1 contacts, however we have identified several potential mechanisms by which it may be possible to read the even and odd rows separately. We demonstrate the attack on a commodity microcontroller made on the 40nm node and show how it can be used to extract significant quantities of sensitive data, such as keys for firmware encryption, in time scales which are very practical for real world exploitation (1 day of sample prep plus a few hours of FIB time) with only a single target device required after initial reconnaissance has been completed on blank devices.

Title: MEDFORM: A Foundation Model for Contrastive Learning of CT Imaging and Clinical Numeric Data in Multi-Cancer Analysis

• Authors: Daeun Jung, Jaehyeok Jang, Sooyoung Jang, Yu Rang Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13277
• Pdf URL: https://arxiv.org/pdf/2501.13277
• Copy Paste: [[2501.13277]] MEDFORM: A Foundation Model for Contrastive Learning of CT Imaging and Clinical Numeric Data in Multi-Cancer Analysis(https://arxiv.org/abs/2501.13277)
• Keywords: robust
• Abstract: Computed tomography (CT) and clinical numeric data are essential modalities for cancer evaluation, but building large-scale multimodal training datasets for developing medical foundation models remains challenging due to the structural complexity of multi-slice CT data and high cost of expert annotation. In this study, we propose MEDFORM, a multimodal pre-training strategy that guides CT image representation learning using complementary information from clinical data for medical foundation model development. MEDFORM efficiently processes CT slice through multiple instance learning (MIL) and adopts a dual pre-training strategy: first pretraining the CT slice feature extractor using SimCLR-based self-supervised learning, then aligning CT and clinical modalities through cross-modal contrastive learning. Our model was pre-trained on three different cancer types: lung cancer (141,171 slices), breast cancer (8,100 slices), colorectal cancer (10,393 slices). The experimental results demonstrated that this dual pre-training strategy improves cancer classification performance and maintains robust performance in few-shot learning scenarios. Code available at this https URL

Title: RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering

• Authors: Yang Bai, Christan Earl Grant, Daisy Zhe Wang
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13297
• Pdf URL: https://arxiv.org/pdf/2501.13297
• Copy Paste: [[2501.13297]] RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering(https://arxiv.org/abs/2501.13297)
• Keywords: generative, large language model
• Abstract: Multi-modal retrieval-augmented Question Answering (MRAQA), integrating text and images, has gained significant attention in information retrieval (IR) and natural language processing (NLP). Traditional ranking methods rely on small encoder-based language models, which are incompatible with modern decoder-based generative large language models (LLMs) that have advanced various NLP tasks. To bridge this gap, we propose RAMQA, a unified framework combining learning-to-rank methods with generative permutation-enhanced ranking techniques. We first train a pointwise multi-modal ranker using LLaVA as the backbone. Then, we apply instruction tuning to train a LLaMA model for re-ranking the top-k documents using an innovative autoregressive multi-task learning approach. Our generative ranking model generates re-ranked document IDs and specific answers from document candidates in various permutations. Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, highlighting the effectiveness of our approach. Code and data are available at: this https URL

Title: Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents

• Authors: Shrinidhi Kumbhar, Venkatesh Mishra, Kevin Coutinho, Divij Handa, Ashif Iquebal, Chitta Baral
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13299
• Pdf URL: https://arxiv.org/pdf/2501.13299
• Copy Paste: [[2501.13299]] Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents(https://arxiv.org/abs/2501.13299)
• Keywords: large language model
• Abstract: Materials discovery and design are essential for advancing technology across various industries by enabling the development of application-specific materials. Recent research has leveraged Large Language Models (LLMs) to accelerate this process. We explore the potential of LLMs to generate viable hypotheses that, once validated, can expedite materials discovery. Collaborating with materials science experts, we curated a novel dataset from recent journal publications, featuring real-world goals, constraints, and methods for designing real-world applications. Using this dataset, we test LLM-based agents that generate hypotheses for achieving given goals under specific constraints. To assess the relevance and quality of these hypotheses, we propose a novel scalable evaluation metric that emulates the process a materials scientist would use to evaluate a hypothesis critically. Our curated dataset, proposed method, and evaluation framework aim to advance future research in accelerating materials discovery and design with LLMs.

Title: Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers

• Authors: Akshit Achara, Anshuman Chhabra
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13302
• Pdf URL: https://arxiv.org/pdf/2501.13302
• Copy Paste: [[2501.13302]] Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers(https://arxiv.org/abs/2501.13302)
• Keywords: robust, fair, large language model
• Abstract: AI Safety Moderation (ASM) classifiers are designed to moderate content on social media platforms and to serve as guardrails that prevent Large Language Models (LLMs) from being fine-tuned on unsafe inputs. Owing to their potential for disparate impact, it is crucial to ensure that these classifiers: (1) do not unfairly classify content belonging to users from minority groups as unsafe compared to those from majority groups and (2) that their behavior remains robust and consistent across similar inputs. In this work, we thus examine the fairness and robustness of four widely-used, closed-source ASM classifiers: OpenAI Moderation API, Perspective API, Google Cloud Natural Language (GCNL) API, and Clarifai API. We assess fairness using metrics such as demographic parity and conditional statistical parity, comparing their performance against ASM models and a fair-only baseline. Additionally, we analyze robustness by testing the classifiers' sensitivity to small and natural input perturbations. Our findings reveal potential fairness and robustness gaps, highlighting the need to mitigate these issues in future versions of these models.

Title: From Cross-Modal to Mixed-Modal Visible-Infrared Re-Identification

• Authors: Mahdi Alehdaghi, Rajarshi Bhattacharya, Pourya Shamsolmoali, Rafael M. O. Cruz, Eric Granger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13307
• Pdf URL: https://arxiv.org/pdf/2501.13307
• Copy Paste: [[2501.13307]] From Cross-Modal to Mixed-Modal Visible-Infrared Re-Identification(https://arxiv.org/abs/2501.13307)
• Keywords: robust
• Abstract: Visible-infrared person re-identification (VI-ReID) aims to match individuals across different camera modalities, a critical task in modern surveillance systems. While current VI-ReID methods focus on cross-modality matching, real-world applications often involve mixed galleries containing both V and I images, where state-of-the-art methods show significant performance limitations due to large domain shifts and low discrimination across mixed modalities. This is because gallery images from the same modality may have lower domain gaps but correspond to different identities. This paper introduces a novel mixed-modal ReID setting, where galleries contain data from both modalities. To address the domain shift among inter-modal and low discrimination capacity in intra-modal matching, we propose the Mixed Modality-Erased and -Related (MixER) method. The MixER learning approach disentangles modality-specific and modality-shared identity information through orthogonal decomposition, modality-confusion, and ID-modality-related objectives. MixER enhances feature robustness across modalities, improving cross-modal and mixed-modal settings performance. Our extensive experiments on the SYSU-MM01, RegDB and LLMC datasets indicate that our approach can provide state-of-the-art results using a single backbone, and showcase the flexibility of our approach in mixed gallery applications.

Title: Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networks

• Authors: Mars Liyao Gao, Jan P. Williams, J. Nathan Kutz
• Subjects: cs.LG, cs.AI, math.DS
• Abstract URL: https://arxiv.org/abs/2501.13329
• Pdf URL: https://arxiv.org/pdf/2501.13329
• Copy Paste: [[2501.13329]] Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networks(https://arxiv.org/abs/2501.13329)
• Keywords: robust
• Abstract: Spatiotemporal modeling of real-world data poses a challenging problem due to inherent high dimensionality, measurement noise, and expensive data collection procedures. In this paper, we present Sparse Identification of Nonlinear Dynamics with SHallow REcurrent Decoder networks (SINDy-SHRED), a method to jointly solve the sensing and model identification problems with simple implementation, efficient computation, and robust performance. SINDy-SHRED uses Gated Recurrent Units (GRUs) to model the temporal sequence of sensor measurements along with a shallow decoder network to reconstruct the full spatiotemporal field from the latent state space using only a few available sensors. Our proposed algorithm introduces a SINDy-based regularization; beginning with an arbitrary latent state space, the dynamics of the latent space progressively converges to a SINDy-class functional, provided the projection remains within the set. In restricting SINDy to a linear model, the architecture produces a Koopman-SHRED model which enforces a linear latent space dynamics. We conduct a systematic experimental study including synthetic PDE data, real-world sensor measurements for sea surface temperature, and direct video data. With no explicit encoder, SINDy-SHRED and Koopman-SHRED enable efficient training with minimal hyperparameter tuning and laptop-level computing; further, it demonstrates robust generalization in a variety of applications with minimal to no hyperparameter adjustments. Finally, the interpretable SINDy and Koopman models of latent state dynamics enables accurate long-term video predictions, achieving state-of-the-art performance and outperforming all baseline methods considered, including Convolutional LSTM, PredRNN, ResNet, and SimVP.

Title: Qrazor: Reliable and effortless 4-bit llm quantization by significant data razoring

• Authors: Dongyoung Lee, Seungkyu Choi, Ik Joon Chang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13331
• Pdf URL: https://arxiv.org/pdf/2501.13331
• Copy Paste: [[2501.13331]] Qrazor: Reliable and effortless 4-bit llm quantization by significant data razoring(https://arxiv.org/abs/2501.13331)
• Keywords: transformer
• Abstract: Large-scale language models (LLMs) have demonstrated outstanding performance in language processing tasks, yet their deployment is often hindered by high memory demands and computational complexity. Although low-bit quantization techniques, such as 4-bit quantization, present a potential solution, they frequently lead to significant accuracy degradation or require substantial effort for such aggressive quantization approaches. To overcome these challenges, we introduce QRazor, a reliable and effortless quantization scheme designed to enable 4-bit quantization for weights, activations, and KV cache in transformer-based LLMs. The scheme involves two main stages: quantization and compression. During the quantization stage, weights, activations, and KV cache values are quantized with wider 8 or 16-bit integers as a basis to achieve nearly identical accuracy to the original full-precision LLM models, using the absolute max scaling. Subsequently, all data are compressed to 4-bit using our proposed significant data razoring (SDR) technique, which retains only the four most salient bits while discarding the others. Furthermore, we present an integer-based arithmetic unit dedicated to QRazor, enabling direct low-precision arithmetic operations without decompressing the SDR data. Despite the reduced quantization effort, QRazor achieves LLM accuracies better or comparable to state-of-the-art 4-bit methods. By also validating the hardware efficiency, our decompression-free arithmetic unit achieves 61.2% and 57.8% reduction in area and power consumption, respectively.

Title: Gradient-Free Adversarial Purification with Diffusion Models

• Authors: Xuelong Dai, Dong Wang, Duan Mingxing, Bin Xiao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.13336
• Pdf URL: https://arxiv.org/pdf/2501.13336
• Copy Paste: [[2501.13336]] Gradient-Free Adversarial Purification with Diffusion Models(https://arxiv.org/abs/2501.13336)
• Keywords: defense, attack, robust, diffusion
• Abstract: Adversarial training and adversarial purification are two effective and practical defense methods to enhance a model's robustness against adversarial attacks. However, adversarial training necessitates additional training, while adversarial purification suffers from low time efficiency. More critically, current defenses are designed under the perturbation-based adversarial threat model, which is ineffective against the recently proposed unrestricted adversarial attacks. In this paper, we propose an effective and efficient adversarial defense method that counters both perturbation-based and unrestricted adversarial attacks. Our defense is inspired by the observation that adversarial attacks are typically located near the decision boundary and are sensitive to pixel changes. To address this, we introduce adversarial anti-aliasing to mitigate adversarial modifications. Additionally, we propose adversarial super-resolution, which leverages prior knowledge from clean datasets to benignly recover images. These approaches do not require additional training and are computationally efficient without calculating gradients. Extensive experiments against both perturbation-based and unrestricted adversarial attacks demonstrate that our defense method outperforms state-of-the-art adversarial purification methods.

Title: Retrievals Can Be Detrimental: A Contrastive Backdoor Attack Paradigm on Retrieval-Augmented Diffusion Models

• Authors: Hao Fang, Xiaohang Sui, Hongyao Yu, Jiawei Kong, Sijin Yu, Bin Chen, Hao Wu, Shu-Tao Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13340
• Pdf URL: https://arxiv.org/pdf/2501.13340
• Copy Paste: [[2501.13340]] Retrievals Can Be Detrimental: A Contrastive Backdoor Attack Paradigm on Retrieval-Augmented Diffusion Models(https://arxiv.org/abs/2501.13340)
• Keywords: security, attack, diffusion, generative
• Abstract: Diffusion models (DMs) have recently demonstrated remarkable generation capability. However, their training generally requires huge computational resources and large-scale datasets. To solve these, recent studies empower DMs with the advanced Retrieval-Augmented Generation (RAG) technique and propose retrieval-augmented diffusion models (RDMs). By incorporating rich knowledge from an auxiliary database, RAG enhances diffusion models' generation and generalization ability while significantly reducing model parameters. Despite the great success, RAG may introduce novel security issues that warrant further investigation. In this paper, we reveal that the RDM is susceptible to backdoor attacks by proposing a multimodal contrastive attack approach named BadRDM. Our framework fully considers RAG's characteristics and is devised to manipulate the retrieved items for given text triggers, thereby further controlling the generated contents. Specifically, we first insert a tiny portion of images into the retrieval database as target toxicity surrogates. Subsequently, a malicious variant of contrastive learning is adopted to inject backdoors into the retriever, which builds shortcuts from triggers to the toxicity surrogates. Furthermore, we enhance the attacks through novel entropy-based selection and generative augmentation strategies that can derive better toxicity surrogates. Extensive experiments on two mainstream tasks demonstrate the proposed BadRDM achieves outstanding attack effects while preserving the model's benign utility.

Title: Multi-aspect Knowledge Distillation with Large Language Model

• Authors: Taegyeong Lee, Jinsik Bang, Soyeong Kwon, Taehwan Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13341
• Pdf URL: https://arxiv.org/pdf/2501.13341
• Copy Paste: [[2501.13341]] Multi-aspect Knowledge Distillation with Large Language Model(https://arxiv.org/abs/2501.13341)
• Keywords: large language model
• Abstract: Recent advancements in deep learning have significantly improved performance on computer vision tasks. Previous image classification methods primarily modify model architectures or add features, and they optimize models using cross-entropy loss on class logits. Since they focus on classifying images with considering class labels, these methods may struggle to learn various \emph{aspects} of classes (e.g., natural positions and shape changes). Rethinking the previous approach from a novel view, we propose a multi-aspect knowledge distillation method using Multimodal Large Language Models (MLLMs). Our approach involves: 1) querying Large Language Model with multi-aspect questions relevant to the knowledge we want to transfer to the model, 2) extracting corresponding logits from MLLM, and 3) expanding the model's output dimensions to distill these multi-aspect logits. We then apply cross-entropy loss to class logits and binary cross-entropy loss to multi-aspect logits. Through our method, the model can learn not only the knowledge about visual aspects but also the abstract and complex aspects that require a deeper understanding. We primarily apply our method to image classification, and to explore the potential for extending our model, we expand it to other tasks, such as object detection. In all experimental results, our method improves the performance of the baselines. Additionally, we analyze the effect of multi-aspect knowledge distillation. These results demonstrate that our method can transfer knowledge about various aspects to the model and the aspect knowledge can enhance model performance in computer vision tasks. This paper demonstrates the great potential of multi-aspect knowledge distillation, and we believe it offers a promising direction for future research in computer vision and beyond.

Title: YOLOSCM: An improved YOLO algorithm for cars detection

• Authors: Changhui Deng, Lieyang Chen, Shinan Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13343
• Pdf URL: https://arxiv.org/pdf/2501.13343
• Copy Paste: [[2501.13343]] YOLOSCM: An improved YOLO algorithm for cars detection(https://arxiv.org/abs/2501.13343)
• Keywords: segmentation
• Abstract: Detecting objects in urban traffic images presents considerable difficulties because of the following reasons: 1) These images are typically immense in size, encompassing millions or even hundreds of millions of pixels, yet computational resources are constrained. 2) The small size of vehicles in certain scenarios leads to insufficient information for accurate detection. 3) The uneven distribution of vehicles causes inefficient use of computational resources. To address these issues, we propose YOLOSCM (You Only Look Once with Segmentation Clustering Module), an efficient and effective framework. To address the challenges of large-scale images and the non-uniform distribution of vehicles, we propose a Segmentation Clustering Module (SCM). This module adaptively identifies clustered regions, enabling the model to focus on these areas for more precise detection. Additionally, we propose a new training strategy to optimize the detection of small vehicles and densely packed targets in complex urban traffic scenes. We perform extensive experiments on urban traffic datasets to demonstrate the effectiveness and superiority of our proposed approach.

Title: One Fits All: General Mobility Trajectory Modeling via Masked Conditional Diffusion

• Authors: Qingyue Long, Can Rong, Huandong Wang, Yong Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13347
• Pdf URL: https://arxiv.org/pdf/2501.13347
• Copy Paste: [[2501.13347]] One Fits All: General Mobility Trajectory Modeling via Masked Conditional Diffusion(https://arxiv.org/abs/2501.13347)
• Keywords: diffusion
• Abstract: Trajectory data play a crucial role in many applications, ranging from network optimization to urban planning. Existing studies on trajectory data are task-specific, and their applicability is limited to the specific tasks on which they have been trained, such as generation, recovery, or prediction. However, the potential of a unified model has not yet been fully explored in trajectory modeling. Although various trajectory tasks differ in inputs, outputs, objectives, and conditions, they share common mobility patterns. Based on these common patterns, we can construct a general framework that enables a single model to address different tasks. However, building a trajectory task-general framework faces two critical challenges: 1) the diversity in the formats of different tasks and 2) the complexity of the conditions imposed on different tasks. In this work, we propose a general trajectory modeling framework via masked conditional diffusion (named GenMove). Specifically, we utilize mask conditions to unify diverse formats. To adapt to complex conditions associated with different tasks, we utilize historical trajectory data to obtain contextual trajectory embeddings, which include rich contexts such as spatiotemporal characteristics and user preferences. Integrating the contextual trajectory embedding into diffusion models through a classifier-free guidance approach allows the model to flexibly adjust its outputs based on different conditions. Extensive experiments on mainstream tasks demonstrate that our model significantly outperforms state-of-the-art baselines, with the highest performance improvement exceeding 13% in generation tasks.

Title: MSF: Efficient Diffusion Model Via Multi-Scale Latent Factorize

• Authors: Haohang Xu, Longyu Chen, Shuangrui Ding, Yilin Gao, Dongsheng Jiang, Yin Li, Shugong Xu, Junqing Yu, Wei Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13349
• Pdf URL: https://arxiv.org/pdf/2501.13349
• Copy Paste: [[2501.13349]] MSF: Efficient Diffusion Model Via Multi-Scale Latent Factorize(https://arxiv.org/abs/2501.13349)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion-based generative models have achieved remarkable progress in visual content generation. However, traditional diffusion models directly denoise the entire image from noisy inputs, disregarding the hierarchical structure present in visual signals. This method is computationally intensive, especially for high-resolution image generation. Signal processing often leverages hierarchical decompositions; for instance, Fourier analysis decomposes signals by frequency, while wavelet analysis captures localized frequency components, reflecting both spatial and frequency information simultaneously. Inspired by these principles, we propose a multiscale diffusion framework that generates hierarchical visual representations, which are subsequently integrated to form the final output. The diffusion model target, whether raw RGB pixels or latent features from a Variational Autoencoder, s divided into multiple components that each capture distinct spatial levels. The low-resolution component contains the primary informative signal, while higher-resolution components add high-frequency details, such as texture. This approach divides image generation into two stages: producing a low-resolution base signal, followed by a high-resolution residual signal. Both stages can be effectively modeled using simpler, lightweight transformer architectures compared to full-resolution generation. This decomposition is conceptually similar to wavelet decomposition but offers a more streamlined and intuitive design. Our method, termed MSF(short for Multi-Scale Factorization), achieves an FID of 2.2 and an IS of 255.4 on the ImageNet 256x256 benchmark, reducing computational costs by 50% compared to baseline methods.

Title: 50 Shades of Deceptive Patterns: A Unified Taxonomy, Multimodal Detection, and Security Implications

• Authors: Zewei Shi, Ruoxi Sun, Jieshan Chen, Jiamou Sun, Minhui Xue, Yansong Gao, Feng Liu, Xingliang Yuan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13351
• Pdf URL: https://arxiv.org/pdf/2501.13351
• Copy Paste: [[2501.13351]] 50 Shades of Deceptive Patterns: A Unified Taxonomy, Multimodal Detection, and Security Implications(https://arxiv.org/abs/2501.13351)
• Keywords: security, privacy, large language model
• Abstract: Deceptive patterns (DPs) are user interface designs deliberately crafted to manipulate users into unintended decisions, often by exploiting cognitive biases for the benefit of companies or services. While numerous studies have explored ways to identify these deceptive patterns, many existing solutions require significant human intervention and struggle to keep pace with the evolving nature of deceptive designs. To address these challenges, we expanded the deceptive pattern taxonomy from security and privacy perspectives, refining its categories and scope. We created a comprehensive dataset of deceptive patterns by integrating existing small-scale datasets with new samples, resulting in 6,725 images and 10,421 DP instances from mobile apps and websites. We then developed DPGuard, a novel automatic tool leveraging commercial multimodal large language models (MLLMs) for deceptive pattern detection. Experimental results show that DPGuard outperforms state-of-the-art methods. Finally, we conducted an extensive empirical evaluation on 2,000 popular mobile apps and websites, revealing that 23.61% of mobile screenshots and 47.27% of website screenshots feature at least one deceptive pattern instance. Through four unexplored case studies that inform security implications, we highlight the critical importance of the unified taxonomy in addressing the growing challenges of Internet deception.

Title: Contrast: A Hybrid Architecture of Transformers and State Space Models for Low-Level Vision

• Authors: Aman Urumbekov, Zheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13353
• Pdf URL: https://arxiv.org/pdf/2501.13353
• Copy Paste: [[2501.13353]] Contrast: A Hybrid Architecture of Transformers and State Space Models for Low-Level Vision(https://arxiv.org/abs/2501.13353)
• Keywords: transformer
• Abstract: Transformers have become increasingly popular for image super-resolution (SR) tasks due to their strong global context modeling capabilities. However, their quadratic computational complexity necessitates the use of window-based attention mechanisms, which restricts the receptive field and limits effective context expansion. Recently, the Mamba architecture has emerged as a promising alternative with linear computational complexity, allowing it to avoid window mechanisms and maintain a large receptive field. Nevertheless, Mamba faces challenges in handling long-context dependencies when high pixel-level precision is required, as in SR tasks. This is due to its hidden state mechanism, which can compress and store a substantial amount of context but only in an approximate manner, leading to inaccuracies that transformers do not suffer from. In this paper, we propose \textbf{Contrast}, a hybrid SR model that combines \textbf{Con}volutional, \textbf{Tra}nsformer, and \textbf{St}ate Space components, effectively blending the strengths of transformers and Mamba to address their individual limitations. By integrating transformer and state space mechanisms, \textbf{Contrast} compensates for the shortcomings of each approach, enhancing both global context modeling and pixel-level accuracy. We demonstrate that combining these two architectures allows us to mitigate the problems inherent in each, resulting in improved performance on image super-resolution tasks.

Title: A light-weight model to generate NDWI from Sentinel-1

• Authors: Saleh Sakib Ahmed, Saifur Rahman Jony, Md. Toufikuzzaman, Saifullah Sayed, Rashed Uz Zzaman, Sara Nowreen, M. Sohel Rahman
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.13357
• Pdf URL: https://arxiv.org/pdf/2501.13357
• Copy Paste: [[2501.13357]] A light-weight model to generate NDWI from Sentinel-1(https://arxiv.org/abs/2501.13357)
• Keywords: robust, segmentation
• Abstract: The use of Sentinel-2 images to compute Normalized Difference Water Index (NDWI) has many applications, including water body area detection. However, cloud cover poses significant challenges in this regard, which hampers the effectiveness of Sentinel-2 images in this context. In this paper, we present a deep learning model that can generate NDWI given Sentinel-1 images, thereby overcoming this cloud barrier. We show the effectiveness of our model, where it demonstrates a high accuracy of 0.9134 and an AUC of 0.8656 to predict the NDWI. Additionally, we observe promising results with an R2 score of 0.4984 (for regressing the NDWI values) and a Mean IoU of 0.4139 (for the underlying segmentation task). In conclusion, our model offers a first and robust solution for generating NDWI images directly from Sentinel-1 images and subsequent use for various applications even under challenging conditions such as cloud cover and nighttime.

Title: False Sense of Security on Protected Wi-Fi Networks

• Authors: Yong Zhi Lim, Hazmei Bin Abdul Rahman, Biplab Sikdar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13363
• Pdf URL: https://arxiv.org/pdf/2501.13363
• Copy Paste: [[2501.13363]] False Sense of Security on Protected Wi-Fi Networks(https://arxiv.org/abs/2501.13363)
• Keywords: secure, security, protect
• Abstract: The Wi-Fi technology (IEEE 802.11) was introduced in 1997. With the increasing use and deployment of such networks, their security has also attracted considerable attention. Current Wi-Fi networks use WPA2 (Wi-Fi Protected Access 2) for security (authentication and encryption) between access points and clients. According to the IEEE 802.11i-2004 standard, wireless networks secured with WPA2-PSK (Pre-Shared Key) are required to be protected with a passphrase between 8 to 63 ASCII characters. However, a poorly chosen passphrase significantly reduces the effectiveness of both WPA2 and WPA3-Personal Transition Mode. The objective of this paper is to empirically evaluate password choices in the wild and evaluate weakness in current common practices. We collected a total of 3,352 password hashes from Wi-Fi access points and determine the passphrases that were protecting them. We then analyze these passwords to investigate the impact of user's behavior and preference for convenience on passphrase strength in secured private Wi-Fi networks in Singapore. We characterized the predictability of passphrases that use the minimum required length of 8 numeric or alphanumeric characters, and/or symbols stipulated in wireless security standards, and the usage of default passwords, and found that 16 percent of the passwords show such behavior. Our results also indicate the prevalence of the use of default passwords by hardware manufacturers. We correlate our results with our findings and recommend methods that will improve the overall security and future of our Wi-Fi networks.

Title: A review on development of eco-friendly filters in Nepal for use in cigarettes and masks and Air Pollution Analysis with Machine Learning and SHAP Interpretability

• Authors: Bishwash Paneru, Biplov Paneru, Tanka Mukhiya, Khem Narayan Poudyal
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13369
• Pdf URL: https://arxiv.org/pdf/2501.13369
• Copy Paste: [[2501.13369]] A review on development of eco-friendly filters in Nepal for use in cigarettes and masks and Air Pollution Analysis with Machine Learning and SHAP Interpretability(https://arxiv.org/abs/2501.13369)
• Keywords: defense, interpretability
• Abstract: In Nepal, air pollution is a serious public health concern, especially in cities like Kathmandu where particulate matter (PM2.5 and PM10) has a major influence on respiratory health and air quality. The Air Quality Index (AQI) is predicted in this work using a Random Forest Regressor, and the model's predictions are interpreted using SHAP (SHapley Additive exPlanations) analysis. With the lowest Testing RMSE (0.23) and flawless R2 scores (1.00), CatBoost performs better than other models, demonstrating its greater accuracy and generalization which is cross validated using a nested cross validation approach. NowCast Concentration and Raw Concentration are the most important elements influencing AQI values, according to SHAP research, which shows that the machine learning results are highly accurate. Their significance as major contributors to air pollution is highlighted by the fact that high values of these characteristics significantly raise the AQI. This study investigates the Hydrogen-Alpha (HA) biodegradable filter as a novel way to reduce the related health hazards. With removal efficiency of more than 98% for PM2.5 and 99.24% for PM10, the HA filter offers exceptional defense against dangerous airborne particles. These devices, which are biodegradable face masks and cigarette filters, address the environmental issues associated with traditional filters' non-biodegradable trash while also lowering exposure to air contaminants.

Title: Do as We Do, Not as You Think: the Conformity of Large Language Models

• Authors: Zhiyuan Weng, Guikun Chen, Wenguan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13381
• Pdf URL: https://arxiv.org/pdf/2501.13381
• Copy Paste: [[2501.13381]] Do as We Do, Not as You Think: the Conformity of Large Language Models(https://arxiv.org/abs/2501.13381)
• Keywords: robust, large language model
• Abstract: Recent advancements in large language models (LLMs) revolutionize the field of intelligent agents, enabling collaborative multi-agent systems capable of tackling complex problems across various domains. However, the potential of conformity within these systems, analogous to phenomena like conformity bias and groupthink in human group dynamics, remains largely unexplored, raising concerns about their collective problem-solving capabilities and possible ethical implications. This paper presents a comprehensive study on conformity in LLM-driven multi-agent systems, focusing on three aspects: the existence of conformity, the factors influencing conformity, and potential mitigation strategies. In particular, we introduce BenchForm, a new conformity-oriented benchmark, featuring reasoning-intensive tasks and five distinct interaction protocols designed to probe LLMs' behavior in collaborative scenarios. Several representative LLMs are evaluated on BenchForm, using metrics such as conformity rate and independence rate to quantify conformity's impact. Our analysis delves into factors influencing conformity, including interaction time and majority size, and examines how the subject agent rationalizes its conforming behavior. Furthermore, we explore two strategies to mitigate conformity effects, i.e., developing enhanced personas and implementing a reflection mechanism. Several interesting findings regarding LLMs' conformity are derived from empirical results and case studies. We hope that these insights can pave the way for more robust and ethically-aligned collaborative AI systems. Our benchmark and code are available at BenchForm.

Title: AEON: Adaptive Estimation of Instance-Dependent In-Distribution and Out-of-Distribution Label Noise for Robust Learning

• Authors: Arpit Garg, Cuong Nguyen, Rafael Felix, Yuyuan Liu, Thanh-Toan Do, Gustavo Carneiro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13389
• Pdf URL: https://arxiv.org/pdf/2501.13389
• Copy Paste: [[2501.13389]] AEON: Adaptive Estimation of Instance-Dependent In-Distribution and Out-of-Distribution Label Noise for Robust Learning(https://arxiv.org/abs/2501.13389)
• Keywords: robust
• Abstract: Robust training with noisy labels is a critical challenge in image classification, offering the potential to reduce reliance on costly clean-label datasets. Real-world datasets often contain a mix of in-distribution (ID) and out-of-distribution (OOD) instance-dependent label noise, a challenge that is rarely addressed simultaneously by existing methods and is further compounded by the lack of comprehensive benchmarking datasets. Furthermore, even though current noisy-label learning approaches attempt to find noisy-label samples during training, these methods do not aim to estimate ID and OOD noise rates to promote their effectiveness in the selection of such noisy-label samples, and they are often represented by inefficient multi-stage learning algorithms. We propose the Adaptive Estimation of Instance-Dependent In-Distribution and Out-of-Distribution Label Noise (AEON) approach to address these research gaps. AEON is an efficient one-stage noisy-label learning methodology that dynamically estimates instance-dependent ID and OOD label noise rates to enhance robustness to complex noise settings. Additionally, we introduce a new benchmark reflecting real-world ID and OOD noise scenarios. Experiments demonstrate that AEON achieves state-of-the-art performance on both synthetic and real-world datasets

Title: Can Large Language Models Understand Preferences in Personalized Recommendation?

• Authors: Zhaoxuan Tan, Zinan Zeng, Qingkai Zeng, Zhenyu Wu, Zheyuan Liu, Fengran Mo, Meng Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13391
• Pdf URL: https://arxiv.org/pdf/2501.13391
• Copy Paste: [[2501.13391]] Can Large Language Models Understand Preferences in Personalized Recommendation?(https://arxiv.org/abs/2501.13391)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) excel in various tasks, including personalized recommendations. Existing evaluation methods often focus on rating prediction, relying on regression errors between actual and predicted ratings. However, user rating bias and item quality, two influential factors behind rating scores, can obscure personal preferences in user-item pair data. To address this, we introduce PerRecBench, disassociating the evaluation from these two factors and assessing recommendation techniques on capturing the personal preferences in a grouped ranking manner. We find that the LLM-based recommendation techniques that are generally good at rating prediction fail to identify users' favored and disfavored items when the user rating bias and item quality are eliminated by grouping users. With PerRecBench and 19 LLMs, we find that while larger models generally outperform smaller ones, they still struggle with personalized recommendation. Our findings reveal the superiority of pairwise and listwise ranking approaches over pointwise ranking, PerRecBench's low correlation with traditional regression metrics, the importance of user profiles, and the role of pretraining data distributions. We further explore three supervised fine-tuning strategies, finding that merging weights from single-format training is promising but improving LLMs' understanding of user preferences remains an open research problem. Code and data are available at this https URL

Title: Towards Intelligent Design: A Self-driven Framework for Collocated Clothing Synthesis Leveraging Fashion Styles and Textures

• Authors: Minglong Dong, Dongliang Zhou, Jianghong Ma, Haijun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13396
• Pdf URL: https://arxiv.org/pdf/2501.13396
• Copy Paste: [[2501.13396]] Towards Intelligent Design: A Self-driven Framework for Collocated Clothing Synthesis Leveraging Fashion Styles and Textures(https://arxiv.org/abs/2501.13396)
• Keywords: generative
• Abstract: Collocated clothing synthesis (CCS) has emerged as a pivotal topic in fashion technology, primarily concerned with the generation of a clothing item that harmoniously matches a given item. However, previous investigations have relied on using paired outfits, such as a pair of matching upper and lower clothing, to train a generative model for achieving this task. This reliance on the expertise of fashion professionals in the construction of such paired outfits has engendered a laborious and time-intensive process. In this paper, we introduce a new self-driven framework, named style- and texture-guided generative network (ST-Net), to synthesize collocated clothing without the necessity for paired outfits, leveraging self-supervised learning. ST-Net is designed to extrapolate fashion compatibility rules from the style and texture attributes of clothing, using a generative adversarial network. To facilitate the training and evaluation of our model, we have constructed a large-scale dataset specifically tailored for unsupervised CCS. Extensive experiments substantiate that our proposed method outperforms the state-of-the-art baselines in terms of both visual authenticity and fashion compatibility.

Title: YOLOv8 to YOLO11: A Comprehensive Architecture In-depth Comparative Review

• Authors: Priyanto Hidayatullah, Nurjannah Syakrani, Muhammad Rizqi Sholahuddin, Trisna Gelar, Refdinal Tubagus
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13400
• Pdf URL: https://arxiv.org/pdf/2501.13400
• Copy Paste: [[2501.13400]] YOLOv8 to YOLO11: A Comprehensive Architecture In-depth Comparative Review(https://arxiv.org/abs/2501.13400)
• Keywords: extraction
• Abstract: In the field of deep learning-based computer vision, YOLO is revolutionary. With respect to deep learning models, YOLO is also the one that is evolving the most rapidly. Unfortunately, not every YOLO model possesses scholarly publications. Moreover, there exists a YOLO model that lacks a publicly accessible official architectural diagram. Naturally, this engenders challenges, such as complicating the understanding of how the model operates in practice. Furthermore, the review articles that are presently available do not delve into the specifics of each model. The objective of this study is to present a comprehensive and in-depth architecture comparison of the four most recent YOLO models, specifically YOLOv8 through YOLO11, thereby enabling readers to quickly grasp not only how each model functions, but also the distinctions between them. To analyze each YOLO version's architecture, we meticulously examined the relevant academic papers, documentation, and scrutinized the source code. The analysis reveals that while each version of YOLO has improvements in architecture and feature extraction, certain blocks remain unchanged. The lack of scholarly publications and official diagrams presents challenges for understanding the model's functionality and future enhancement. Future developers are encouraged to provide these resources.

Title: M3PT: A Transformer for Multimodal, Multi-Party Social Signal Prediction with Person-aware Blockwise Attention

• Authors: Yiming Tang, Abrar Anwar, Jesse Thomason
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.13416
• Pdf URL: https://arxiv.org/pdf/2501.13416
• Copy Paste: [[2501.13416]] M3PT: A Transformer for Multimodal, Multi-Party Social Signal Prediction with Person-aware Blockwise Attention(https://arxiv.org/abs/2501.13416)
• Keywords: transformer
• Abstract: Understanding social signals in multi-party conversations is important for human-robot interaction and artificial social intelligence. Multi-party interactions include social signals like body pose, head pose, speech, and context-specific activities like acquiring and taking bites of food when dining. Incorporating all the multimodal signals in a multi-party interaction is difficult, and past work tends to build task-specific models for predicting social signals. In this work, we address the challenge of predicting multimodal social signals in multi-party settings in a single model. We introduce M3PT, a causal transformer architecture with modality and temporal blockwise attention masking which allows for the simultaneous processing of multiple social cues across multiple participants and their temporal interactions. This approach better captures social dynamics over time by considering longer horizons of social signals between individuals. We train and evaluate our unified model on the Human-Human Commensality Dataset (HHCD), and demonstrate that using multiple modalities improves bite timing and speaking status prediction. Source code: this https URL

Title: Auto-Prompting SAM for Weakly Supervised Landslide Extraction

• Authors: Jian Wang, Xiaokang Zhang, Xianping Ma, Weikang Yu, Pedram Ghamisi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13426
• Pdf URL: https://arxiv.org/pdf/2501.13426
• Copy Paste: [[2501.13426]] Auto-Prompting SAM for Weakly Supervised Landslide Extraction(https://arxiv.org/abs/2501.13426)
• Keywords: extraction, segmentation
• Abstract: Weakly supervised landslide extraction aims to identify landslide regions from remote sensing data using models trained with weak labels, particularly image-level labels. However, it is often challenged by the imprecise boundaries of the extracted objects due to the lack of pixel-wise supervision and the properties of landslide objects. To tackle these issues, we propose a simple yet effective method by auto-prompting the Segment Anything Model (SAM), i.e., APSAM. Instead of depending on high-quality class activation maps (CAMs) for pseudo-labeling or fine-tuning SAM, our method directly yields fine-grained segmentation masks from SAM inference through prompt engineering. Specifically, it adaptively generates hybrid prompts from the CAMs obtained by an object localization network. To provide sufficient information for SAM prompting, an adaptive prompt generation (APG) algorithm is designed to fully leverage the visual patterns of CAMs, enabling the efficient generation of pseudo-masks for landslide extraction. These informative prompts are able to identify the extent of landslide areas (box prompts) and denote the centers of landslide objects (point prompts), guiding SAM in landslide segmentation. Experimental results on high-resolution aerial and satellite datasets demonstrate the effectiveness of our method, achieving improvements of at least 3.0\% in F1 score and 3.69\% in IoU compared to other state-of-the-art methods. The source codes and datasets will be available at this https URL.

Title: Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models

• Authors: Bo Gao, Michael W. Spratling
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13428
• Pdf URL: https://arxiv.org/pdf/2501.13428
• Copy Paste: [[2501.13428]] Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models(https://arxiv.org/abs/2501.13428)
• Keywords: large language model
• Abstract: Large language models have achieved remarkable success in recent years, primarily due to the implementation of self-attention mechanisms. However, traditional Softmax attention suffers from numerical instability and reduced performance as the length of inference tokens increases. This paper addresses these issues by decomposing the Softmax operation into a non-linear transformation and the $l_1$-norm. We identify the latter as essential for maintaining model performance. By replacing the non-linear transformation with the Softplus activation function and introducing a dynamic length scale factor for different token lengths based on invariance entropy, we create a novel attention mechanism with performance better than conventional Softmax attention across various inference lengths. To further improve the length extrapolation ability of the proposed attention mechanism, we introduce a re-weighting mechanism that amplifies significant attention weights while diminishing weaker ones, enabling the model to concentrate more effectively on relevant tokens. When combined with our proposed attention mechanism, this approach demonstrates significant promise in managing longer sequences, maintaining nearly constant validation loss even at 16$\times$ the training token length while ensuring numerical stability. Our code is available at: this https URL.

Title: GC-ConsFlow: Leveraging Optical Flow Residuals and Global Context for Robust Deepfake Detection

• Authors: Jiaxin Chen, Miao Hu, Dengyong Zhang, Jingyang Meng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13435
• Pdf URL: https://arxiv.org/pdf/2501.13435
• Copy Paste: [[2501.13435]] GC-ConsFlow: Leveraging Optical Flow Residuals and Global Context for Robust Deepfake Detection(https://arxiv.org/abs/2501.13435)
• Keywords: robust, extraction
• Abstract: The rapid development of Deepfake technology has enabled the generation of highly realistic manipulated videos, posing severe social and ethical challenges. Existing Deepfake detection methods primarily focused on either spatial or temporal inconsistencies, often neglecting the interplay between the two or suffering from interference caused by natural facial motions. To address these challenges, we propose the global context consistency flow (GC-ConsFlow), a novel dual-stream framework that effectively integrates spatial and temporal features for robust Deepfake detection. The global grouped context aggregation module (GGCA), integrated into the global context-aware frame flow stream (GCAF), enhances spatial feature extraction by aggregating grouped global context information, enabling the detection of subtle, spatial artifacts within frames. The flow-gradient temporal consistency stream (FGTC), rather than directly modeling the residuals, it is used to improve the robustness of temporal feature extraction against the inconsistency introduced by unnatural facial motion using optical flow residuals and gradient-based features. By combining these two streams, GC-ConsFlow demonstrates the effectiveness and robustness in capturing complementary spatiotemporal forgery traces. Extensive experiments show that GC-ConsFlow outperforms existing state-of-the-art methods in detecting Deepfake videos under various compression scenarios.

Title: MultiDreamer3D: Multi-concept 3D Customization with Concept-Aware Diffusion Guidance

• Authors: Wooseok Song, Seunggyu Chang, Jaejun Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13449
• Pdf URL: https://arxiv.org/pdf/2501.13449
• Copy Paste: [[2501.13449]] MultiDreamer3D: Multi-concept 3D Customization with Concept-Aware Diffusion Guidance(https://arxiv.org/abs/2501.13449)
• Keywords: diffusion
• Abstract: While single-concept customization has been studied in 3D, multi-concept customization remains largely unexplored. To address this, we propose MultiDreamer3D that can generate coherent multi-concept 3D content in a divide-and-conquer manner. First, we generate 3D bounding boxes using an LLM-based layout controller. Next, a selective point cloud generator creates coarse point clouds for each concept. These point clouds are placed in the 3D bounding boxes and initialized into 3D Gaussian Splatting with concept labels, enabling precise identification of concept attributions in 2D projections. Finally, we refine 3D Gaussians via concept-aware interval score matching, guided by concept-aware diffusion. Our experimental results show that MultiDreamer3D not only ensures object presence and preserves the distinct identities of each concept but also successfully handles complex cases such as property change or interaction. To the best of our knowledge, we are the first to address the multi-concept customization in 3D.

Title: EchoVideo: Identity-Preserving Human Video Generation by Multimodal Feature Fusion

• Authors: Jiangchuan Wei, Shiyue Yan, Wenfeng Lin, Boyuan Liu, Renjie Chen, Mingyu Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13452
• Pdf URL: https://arxiv.org/pdf/2501.13452
• Copy Paste: [[2501.13452]] EchoVideo: Identity-Preserving Human Video Generation by Multimodal Feature Fusion(https://arxiv.org/abs/2501.13452)
• Keywords: robust
• Abstract: Recent advancements in video generation have significantly impacted various downstream applications, particularly in identity-preserving video generation (IPT2V). However, existing methods struggle with "copy-paste" artifacts and low similarity issues, primarily due to their reliance on low-level facial image information. This dependence can result in rigid facial appearances and artifacts reflecting irrelevant details. To address these challenges, we propose EchoVideo, which employs two key strategies: (1) an Identity Image-Text Fusion Module (IITF) that integrates high-level semantic features from text, capturing clean facial identity representations while discarding occlusions, poses, and lighting variations to avoid the introduction of artifacts; (2) a two-stage training strategy, incorporating a stochastic method in the second phase to randomly utilize shallow facial information. The objective is to balance the enhancements in fidelity provided by shallow features while mitigating excessive reliance on them. This strategy encourages the model to utilize high-level features during training, ultimately fostering a more robust representation of facial identities. EchoVideo effectively preserves facial identities and maintains full-body integrity. Extensive experiments demonstrate that it achieves excellent results in generating high-quality, controllability and fidelity videos.

Title: Spurious Forgetting in Continual Learning of Language Models

• Authors: Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13453
• Pdf URL: https://arxiv.org/pdf/2501.13453
• Copy Paste: [[2501.13453]] Spurious Forgetting in Continual Learning of Language Models(https://arxiv.org/abs/2501.13453)
• Keywords: robust, large language model
• Abstract: Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

Title: Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer

• Authors: Jia Gao, Guiran Liu, Binrong Zhu, Shicheng Zhou, Hongye Zheng, Xiaoxuan Liao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13467
• Pdf URL: https://arxiv.org/pdf/2501.13467
• Copy Paste: [[2501.13467]] Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer(https://arxiv.org/abs/2501.13467)
• Keywords: transformer
• Abstract: This paper studies a text classification algorithm based on an improved Transformer to improve the performance and efficiency of the model in text classification tasks. Aiming at the shortcomings of the traditional Transformer model in capturing deep semantic relationships and optimizing computational complexity, this paper introduces a multi-level attention mechanism and a contrastive learning strategy. The multi-level attention mechanism effectively models the global semantics and local features in the text by combining global attention with local attention; the contrastive learning strategy enhances the model's ability to distinguish between different categories by constructing positive and negative sample pairs while improving the classification effect. In addition, in order to improve the training and inference efficiency of the model on large-scale text data, this paper designs a lightweight module to optimize the feature transformation process and reduce the computational cost. Experimental results on the dataset show that the improved Transformer model outperforms the comparative models such as BiLSTM, CNN, standard Transformer, and BERT in terms of classification accuracy, F1 score, and recall rate, showing stronger semantic representation ability and generalization performance. The method proposed in this paper provides a new idea for algorithm optimization in the field of text classification and has good application potential and practical value. Future work will focus on studying the performance of this model in multi-category imbalanced datasets and cross-domain tasks and explore the integration wi

Title: Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

• Authors: Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, Huchuan Lu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13468
• Pdf URL: https://arxiv.org/pdf/2501.13468
• Copy Paste: [[2501.13468]] Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge(https://arxiv.org/abs/2501.13468)
• Keywords: robust, large language model
• Abstract: Recent advances in Large Language Models (LLMs) have enabled the development of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. To address these issues, we propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. $\StreamChat$ leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time, multi-turn dialogue. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Furthermore, we introduce StreamBench, a versatile benchmark that evaluates streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. Extensive evaluations on StreamBench and other public benchmarks demonstrate that StreamChat significantly outperforms existing state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding. Code is available at StreamChat: this https URL.

Title: Leveraging Textual Anatomical Knowledge for Class-Imbalanced Semi-Supervised Multi-Organ Segmentation

• Authors: Yuliang Gu, Weilun Tsao, Bo Du, Thierry Géraud, Yongchao Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13470
• Pdf URL: https://arxiv.org/pdf/2501.13470
• Copy Paste: [[2501.13470]] Leveraging Textual Anatomical Knowledge for Class-Imbalanced Semi-Supervised Multi-Organ Segmentation(https://arxiv.org/abs/2501.13470)
• Keywords: segmentation
• Abstract: Annotating 3D medical images demands substantial time and expertise, driving the adoption of semi-supervised learning (SSL) for segmentation tasks. However, the complex anatomical structures of organs often lead to significant class imbalances, posing major challenges for deploying SSL in real-world scenarios. Despite the availability of valuable prior information, such as inter-organ relative positions and organ shape priors, existing SSL methods have yet to fully leverage these insights. To address this gap, we propose a novel approach that integrates textual anatomical knowledge (TAK) into the segmentation model. Specifically, we use GPT-4o to generate textual descriptions of anatomical priors, which are then encoded using a CLIP-based model. These encoded priors are injected into the segmentation model as parameters of the segmentation head. Additionally, contrastive learning is employed to enhance the alignment between textual priors and visual features. Extensive experiments demonstrate the superior performance of our method, significantly surpassing state-of-the-art approaches. The source code will be available at: this https URL.

Title: LDR-Net: A Novel Framework for AI-generated Image Detection via Localized Discrepancy Representation

• Authors: JiaXin Chen, Miao Hu, DengYong Zhang, Yun Song, Xin Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13475
• Pdf URL: https://arxiv.org/pdf/2501.13475
• Copy Paste: [[2501.13475]] LDR-Net: A Novel Framework for AI-generated Image Detection via Localized Discrepancy Representation(https://arxiv.org/abs/2501.13475)
• Keywords: diffusion, generative
• Abstract: With the rapid advancement of generative models, the visual quality of generated images has become nearly indistinguishable from the real ones, posing challenges to content authenticity verification. Existing methods for detecting AI-generated images primarily focus on specific forgery clues, which are often tailored to particular generative models like GANs or diffusion models. These approaches struggle to generalize across architectures. Building on the observation that generative images often exhibit local anomalies, such as excessive smoothness, blurred textures, and unnatural pixel variations in small regions, we propose the localized discrepancy representation network (LDR-Net), a novel approach for detecting AI-generated images. LDR-Net captures smoothing artifacts and texture irregularities, which are common but often overlooked. It integrates two complementary modules: local gradient autocorrelation (LGA) which models local smoothing anomalies to detect smoothing anomalies, and local variation pattern (LVP) which captures unnatural regularities by modeling the complexity of image patterns. By merging LGA and LVP features, a comprehensive representation of localized discrepancies can be provided. Extensive experiments demonstrate that our LDR-Net achieves state-of-the-art performance in detecting generated images and exhibits satisfactory generalization across unseen generative models. The code will be released upon acceptance of this paper.

Title: Adaptive Few-Shot Learning (AFSL): Tackling Data Scarcity with Stability, Robustness, and Versatility

• Authors: Rishabh Agrawal
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13479
• Pdf URL: https://arxiv.org/pdf/2501.13479
• Copy Paste: [[2501.13479]] Adaptive Few-Shot Learning (AFSL): Tackling Data Scarcity with Stability, Robustness, and Versatility(https://arxiv.org/abs/2501.13479)
• Keywords: robust
• Abstract: Few-shot learning (FSL) enables machine learning models to generalize effectively with minimal labeled data, making it crucial for data-scarce domains such as healthcare, robotics, and natural language processing. Despite its potential, FSL faces challenges including sensitivity to initialization, difficulty in adapting to diverse domains, and vulnerability to noisy datasets. To address these issues, this paper introduces Adaptive Few-Shot Learning (AFSL), a framework that integrates advancements in meta-learning, domain alignment, noise resilience, and multi-modal integration. AFSL consists of four key modules: a Dynamic Stability Module for performance consistency, a Contextual Domain Alignment Module for domain adaptation, a Noise-Adaptive Resilience Module for handling noisy data, and a Multi-Modal Fusion Module for integrating diverse modalities. This work also explores strategies such as task-aware data augmentation, semi-supervised learning, and explainable AI techniques to enhance the applicability and robustness of FSL. AFSL provides scalable, reliable, and impactful solutions for real-world, high-stakes domains.

Title: MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods

• Authors: Zukang Xu, Yuxuan Yue, Xing Hu, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Jiangyong Yu, Chen Xu, Sifan Zhou, Dawei Yang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13484
• Pdf URL: https://arxiv.org/pdf/2501.13484
• Copy Paste: [[2501.13484]] MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods(https://arxiv.org/abs/2501.13484)
• Keywords: transformer
• Abstract: Mamba is an efficient sequence model that rivals Transformers and demonstrates significant potential as a foundational architecture for various tasks. Quantization is commonly used in neural networks to reduce model size and computational latency. However, applying quantization to Mamba remains underexplored, and existing quantization methods, which have been effective for CNN and Transformer models, appear inadequate for Mamba models (e.g., Quarot suffers a 21% accuracy drop on Vim-T$^\dagger$ even under W8A8). We have pioneered the exploration of this issue and identified several key challenges. First, significant outliers are present in gate projections, output projections, and matrix multiplications. Second, Mamba's unique parallel scan further amplifies these outliers, leading to uneven and heavy-tailed data distributions. Third, even with the application of the Hadamard transform, the variance across channels in weights and activations still remains inconsistent. To these ends, we propose MambaQuant, a post-training quantization (PTQ) framework consisting of: 1) Karhunen-Loeve Transformation (KLT) enhanced rotation, rendering the rotation matrix adaptable to diverse channel distributions. 2) Smooth-Fused rotation, which equalizes channel variances and can merge additional parameters into model weights. Experiments show that MambaQuant can quantize both weights and activations into 8-bit with less than 1% accuracy loss for Mamba-based vision and language tasks. To the best of our knowledge, MambaQuant is the first comprehensive PTQ design for the Mamba family, paving the way for further advancements in its application.

Title: RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles

• Authors: Munachiso Nwadike, Zangir Iklassov, Toluwani Aremu, Tatsuya Hiraoka, Velibor Bojkovic, Benjamin Heinzerling, Hilal Alqaubeh, Martin Takáč, Kentaro Inui
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13491
• Pdf URL: https://arxiv.org/pdf/2501.13491
• Copy Paste: [[2501.13491]] RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles(https://arxiv.org/abs/2501.13491)
• Keywords: large language model
• Abstract: We introduce the concept of the self-referencing causal cycle (abbreviated RECALL) - a mechanism that enables large language models (LLMs) to bypass the limitations of unidirectional causality, which underlies a phenomenon known as the reversal curse. When an LLM is prompted with sequential data, it often fails to recall preceding context. For example, when we ask an LLM to recall the line preceding "O say does that star-spangled banner yet wave" in the U.S. National Anthem, it often fails to correctly return "Gave proof through the night that our flag was still there" - this is due to the reversal curse. It occurs because language models such as ChatGPT and Llama generate text based on preceding tokens, requiring facts to be learned and reproduced in a consistent token order. While the reversal curse is often viewed as a limitation, we offer evidence of an alternative view: it is not always an obstacle in practice. We find that RECALL is driven by what we designate as cycle tokens - sequences that connect different parts of the training data, enabling recall of preceding tokens from succeeding ones. Through rigorous probabilistic formalization and controlled experiments, we demonstrate how the cycles they induce influence a model's ability to reproduce information. To facilitate reproducibility, we provide our code and experimental details at this https URL.

Title: Quantized Spike-driven Transformer

• Authors: Xuerui Qiu, Jieyuan Zhang, Wenjie Wei, Honglin Cao, Junsheng Guo, Rui-Jie Zhu, Yimeng Shan, Yang Yang, Malu Zhang, Haizhou Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13492
• Pdf URL: https://arxiv.org/pdf/2501.13492
• Copy Paste: [[2501.13492]] Quantized Spike-driven Transformer(https://arxiv.org/abs/2501.13492)
• Keywords: transformer
• Abstract: Spiking neural networks are emerging as a promising energy-efficient alternative to traditional artificial neural networks due to their spike-driven paradigm. However, recent research in the SNN domain has mainly focused on enhancing accuracy by designing large-scale Transformer structures, which typically rely on substantial computational resources, limiting their deployment on resource-constrained devices. To overcome this challenge, we propose a quantized spike-driven Transformer baseline (QSD-Transformer), which achieves reduced resource demands by utilizing a low bit-width parameter. Regrettably, the QSD-Transformer often suffers from severe performance degradation. In this paper, we first conduct empirical analysis and find that the bimodal distribution of quantized spike-driven self-attention (Q-SDSA) leads to spike information distortion (SID) during quantization, causing significant performance degradation. To mitigate this issue, we take inspiration from mutual information entropy and propose a bi-level optimization strategy to rectify the information distribution in Q-SDSA. Specifically, at the lower level, we introduce an information-enhanced LIF to rectify the information distribution in Q-SDSA. At the upper level, we propose a fine-grained distillation scheme for the QSD-Transformer to align the distribution in Q-SDSA with that in the counterpart ANN. By integrating the bi-level optimization strategy, the QSD-Transformer can attain enhanced energy efficiency without sacrificing its high-performance this http URL instance, when compared to the prior SNN benchmark on ImageNet, the QSD-Transformer achieves 80.3\% top-1 accuracy, accompanied by significant reductions of 6.0$\times$ and 8.1$\times$ in power consumption and model size, respectively. Code is available at this https URL.

Title: GCAD: Anomaly Detection in Multivariate Time Series from the Perspective of Granger Causality

• Authors: Zehao Liu, Mengzhou Gao, Pengfei Jiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13493
• Pdf URL: https://arxiv.org/pdf/2501.13493
• Copy Paste: [[2501.13493]] GCAD: Anomaly Detection in Multivariate Time Series from the Perspective of Granger Causality(https://arxiv.org/abs/2501.13493)
• Keywords: interpretability
• Abstract: Multivariate time series anomaly detection has numerous real-world applications and is being extensively studied. Modeling pairwise correlations between variables is crucial. Existing methods employ learnable graph structures and graph neural networks to explicitly model the spatial dependencies between variables. However, these methods are primarily based on prediction or reconstruction tasks, which can only learn similarity relationships between sequence embeddings and lack interpretability in how graph structures affect time series evolution. In this paper, we designed a framework that models spatial dependencies using interpretable causal relationships and detects anomalies through changes in causal patterns. Specifically, we propose a method to dynamically discover Granger causality using gradients in nonlinear deep predictors and employ a simple sparsification strategy to obtain a Granger causality graph, detecting anomalies from a causal perspective. Experiments on real-world datasets demonstrate that the proposed model achieves more accurate anomaly detection compared to baseline methods.

Title: Propensity-driven Uncertainty Learning for Sample Exploration in Source-Free Active Domain Adaptation

• Authors: Zicheng Pan, Xiaohan Yu, Weichuan Zhang, Yongsheng Gao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13517
• Pdf URL: https://arxiv.org/pdf/2501.13517
• Copy Paste: [[2501.13517]] Propensity-driven Uncertainty Learning for Sample Exploration in Source-Free Active Domain Adaptation(https://arxiv.org/abs/2501.13517)
• Keywords: privacy
• Abstract: Source-free active domain adaptation (SFADA) addresses the challenge of adapting a pre-trained model to new domains without access to source data while minimizing the need for target domain annotations. This scenario is particularly relevant in real-world applications where data privacy, storage limitations, or labeling costs are significant concerns. Key challenges in SFADA include selecting the most informative samples from the target domain for labeling, effectively leveraging both labeled and unlabeled target data, and adapting the model without relying on source domain information. Additionally, existing methods often struggle with noisy or outlier samples and may require impractical progressive labeling during training. To effectively select more informative samples without frequently requesting human annotations, we propose the Propensity-driven Uncertainty Learning (ProULearn) framework. ProULearn utilizes a novel homogeneity propensity estimation mechanism combined with correlation index calculation to evaluate feature-level relationships. This approach enables the identification of representative and challenging samples while avoiding noisy outliers. Additionally, we develop a central correlation loss to refine pseudo-labels and create compact class distributions during adaptation. In this way, ProULearn effectively bridges the domain gap and maximizes adaptation performance. The principles of informative sample selection underlying ProULearn have broad implications beyond SFADA, offering benefits across various deep learning tasks where identifying key data points or features is crucial. Extensive experiments on four benchmark datasets demonstrate that ProULearn outperforms state-of-the-art methods in domain adaptation scenarios.

Title: Text-driven Online Action Detection

• Authors: Manuel Benavent-Lledo, David Mulero-Pérez, David Ortiz-Perez, Jose Garcia-Rodriguez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13518
• Pdf URL: https://arxiv.org/pdf/2501.13518
• Copy Paste: [[2501.13518]] Text-driven Online Action Detection(https://arxiv.org/abs/2501.13518)
• Keywords: transformer
• Abstract: Detecting actions as they occur is essential for applications like video surveillance, autonomous driving, and human-robot interaction. Known as online action detection, this task requires classifying actions in streaming videos, handling background noise, and coping with incomplete actions. Transformer architectures are the current state-of-the-art, yet the potential of recent advancements in computer vision, particularly vision-language models (VLMs), remains largely untapped for this problem, partly due to high computational costs. In this paper, we introduce TOAD: a Text-driven Online Action Detection architecture that supports zero-shot and few-shot learning. TOAD leverages CLIP (Contrastive Language-Image Pretraining) textual embeddings, enabling efficient use of VLMs without significant computational overhead. Our model achieves 82.46% mAP on the THUMOS14 dataset, outperforming existing methods, and sets new baselines for zero-shot and few-shot performance on the THUMOS14 and TVSeries datasets.

Title: Diffusion-based Perceptual Neural Video Compression with Temporal Diffusion Information Reuse

• Authors: Wenzhuo Ma, Zhenzhong Chen
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.13528
• Pdf URL: https://arxiv.org/pdf/2501.13528
• Copy Paste: [[2501.13528]] Diffusion-based Perceptual Neural Video Compression with Temporal Diffusion Information Reuse(https://arxiv.org/abs/2501.13528)
• Keywords: robust, diffusion
• Abstract: Recently, foundational diffusion models have attracted considerable attention in image compression tasks, whereas their application to video compression remains largely unexplored. In this article, we introduce DiffVC, a diffusion-based perceptual neural video compression framework that effectively integrates foundational diffusion model with the video conditional coding paradigm. This framework uses temporal context from previously decoded frame and the reconstructed latent representation of the current frame to guide the diffusion model in generating high-quality results. To accelerate the iterative inference process of diffusion model, we propose the Temporal Diffusion Information Reuse (TDIR) strategy, which significantly enhances inference efficiency with minimal performance loss by reusing the diffusion information from previous frames. Additionally, to address the challenges posed by distortion differences across various bitrates, we propose the Quantization Parameter-based Prompting (QPP) mechanism, which utilizes quantization parameters as prompts fed into the foundational diffusion model to explicitly modulate intermediate features, thereby enabling a robust variable bitrate diffusion-based neural compression framework. Experimental results demonstrate that our proposed solution delivers excellent performance in both perception metrics and visual quality.

Title: Overcoming Support Dilution for Robust Few-shot Semantic Segmentation

• Authors: Wailing Tang, Biqi Yang, Pheng-Ann Heng, Yun-Hui Liu, Chi-Wing Fu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13529
• Pdf URL: https://arxiv.org/pdf/2501.13529
• Copy Paste: [[2501.13529]] Overcoming Support Dilution for Robust Few-shot Semantic Segmentation(https://arxiv.org/abs/2501.13529)
• Keywords: robust, segmentation
• Abstract: Few-shot Semantic Segmentation (FSS) is a challenging task that utilizes limited support images to segment associated unseen objects in query images. However, recent FSS methods are observed to perform worse, when enlarging the number of shots. As the support set enlarges, existing FSS networks struggle to concentrate on the high-contributed supports and could easily be overwhelmed by the low-contributed supports that could severely impair the mask predictions. In this work, we study this challenging issue, called support dilution, our goal is to recognize, select, preserve, and enhance those high-contributed supports in the raw support pool. Technically, our method contains three novel parts. First, we propose a contribution index, to quantitatively estimate if a high-contributed support dilutes. Second, we develop the Symmetric Correlation (SC) module to preserve and enhance the high-contributed support features, minimizing the distraction by the low-contributed features. Third, we design the Support Image Pruning operation, to retrieve a compact and high quality subset by discarding low-contributed supports. We conduct extensive experiments on two FSS benchmarks, COCO-20i and PASCAL-5i, the segmentation results demonstrate the compelling performance of our solution over state-of-the-art FSS approaches. Besides, we apply our solution for online segmentation and real-world segmentation, convincing segmentation results showing the practical ability of our work for real-world demonstrations.

Title: ReasVQA: Advancing VideoQA with Imperfect Reasoning Process

• Authors: Jianxin Liang, Xiaojun Meng, Huishuai Zhang, Yueqian Wang, Jiansheng Wei, Dongyan Zhao
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13536
• Pdf URL: https://arxiv.org/pdf/2501.13536
• Copy Paste: [[2501.13536]] ReasVQA: Advancing VideoQA with Imperfect Reasoning Process(https://arxiv.org/abs/2501.13536)
• Keywords: large language model
• Abstract: Video Question Answering (VideoQA) is a challenging task that requires understanding complex visual and temporal relationships within videos to answer questions accurately. In this work, we introduce \textbf{ReasVQA} (Reasoning-enhanced Video Question Answering), a novel approach that leverages reasoning processes generated by Multimodal Large Language Models (MLLMs) to improve the performance of VideoQA models. Our approach consists of three phases: reasoning generation, reasoning refinement, and learning from reasoning. First, we generate detailed reasoning processes using additional MLLMs, and second refine them via a filtering step to ensure data quality. Finally, we use the reasoning data, which might be in an imperfect form, to guide the VideoQA model via multi-task learning, on how to interpret and answer questions based on a given video. We evaluate ReasVQA on three popular benchmarks, and our results establish new state-of-the-art performance with significant improvements of +2.9 on NExT-QA, +7.3 on STAR, and +5.9 on IntentQA. Our findings demonstrate the supervising benefits of integrating reasoning processes into VideoQA. Further studies validate each component of our method, also with different backbones and MLLMs, and again highlight the advantages of this simple but effective method. We offer a new perspective on enhancing VideoQA performance by utilizing advanced reasoning techniques, setting a new benchmark in this research field.

Title: POPS: From History to Mitigation of DNS Cache Poisoning Attacks

• Authors: Yehuda Afek, Harel Berger, Anat Bremler-Barr
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.13540
• Pdf URL: https://arxiv.org/pdf/2501.13540
• Copy Paste: [[2501.13540]] POPS: From History to Mitigation of DNS Cache Poisoning Attacks(https://arxiv.org/abs/2501.13540)
• Keywords: security, protect, attack, robust
• Abstract: We present a novel yet simple and comprehensive DNS cache POisoning Prevention System (POPS), designed to integrate as a module in Intrusion Prevention Systems (IPS). POPS addresses statistical DNS poisoning attacks, including those documented from 2002 to the present, and offers robust protection against similar future threats. It consists of two main components: a detection module that employs three simple rules, and a mitigation module that leverages the TC flag in the DNS header to enhance security. Once activated, the mitigation module has zero false positives or negatives, correcting any such errors on the side of the detection module. We first analyze POPS against historical DNS services and attacks, showing that it would have mitigated all network-based statistical poisoning attacks, yielding a success rate of only 0.0076% for the adversary. We then simulate POPS on traffic benchmarks (PCAPs) incorporating current potential network-based statistical poisoning attacks, and benign PCAPs; the simulated attacks still succeed with a probability of 0.0076%. This occurs because five malicious packets go through before POPS detects the attack and activates the mitigation module. In addition, POPS completes its task using only 20%-50% of the time required by other tools (e.g., Suricata or Snort), and after examining just 5%-10% as many packets. Furthermore, it successfully identifies DNS cache poisoning attacks-such as fragmentation attacks-that both Suricata and Snort fail to detect, underscoring its superiority in providing comprehensive DNS protection.

Title: LLMs Can Plan Only If We Tell Them

• Authors: Bilgehan Sel, Ruoxi Jia, Ming Jin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13545
• Pdf URL: https://arxiv.org/pdf/2501.13545
• Copy Paste: [[2501.13545]] LLMs Can Plan Only If We Tell Them(https://arxiv.org/abs/2501.13545)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated significant capabilities in natural language processing and reasoning, yet their effectiveness in autonomous planning has been under debate. While existing studies have utilized LLMs with external feedback mechanisms or in controlled environments for planning, these approaches often involve substantial computational and development resources due to the requirement for careful design and iterative backprompting. Moreover, even the most advanced LLMs like GPT-4 struggle to match human performance on standard planning benchmarks, such as the Blocksworld, without additional support. This paper investigates whether LLMs can independently generate long-horizon plans that rival human baselines. Our novel enhancements to Algorithm-of-Thoughts (AoT), which we dub AoT+, help achieve state-of-the-art results in planning benchmarks out-competing prior methods and human baselines all autonomously.

Title: One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt

• Authors: Tao Liu, Kai Wang, Senmao Li, Joost van de Weijer, Fahad Shahbaz Khan, Shiqi Yang, Yaxing Wang, Jian Yang, Ming-Ming Cheng
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13554
• Pdf URL: https://arxiv.org/pdf/2501.13554
• Copy Paste: [[2501.13554]] One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt(https://arxiv.org/abs/2501.13554)
• Keywords: diffusion
• Abstract: Text-to-image generation models can create high-quality images from input prompts. However, they struggle to support the consistent generation of identity-preserving requirements for storytelling. Existing approaches to this problem typically require extensive training in large datasets or additional modifications to the original model architectures. This limits their applicability across different domains and diverse diffusion model configurations. In this paper, we first observe the inherent capability of language models, coined context consistency, to comprehend identity through context with a single prompt. Drawing inspiration from the inherent context consistency, we propose a novel training-free method for consistent text-to-image (T2I) generation, termed "One-Prompt-One-Story" (1Prompt1Story). Our approach 1Prompt1Story concatenates all prompts into a single input for T2I diffusion models, initially preserving character identities. We then refine the generation process using two novel techniques: Singular-Value Reweighting and Identity-Preserving Cross-Attention, ensuring better alignment with the input description for each frame. In our experiments, we compare our method against various existing consistent T2I generation approaches to demonstrate its effectiveness through quantitative metrics and qualitative assessments. Code is available at this https URL.

Title: Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving

• Authors: Lu Wang, Tianyuan Zhang, Yang Qu, Siyuan Liang, Yuwei Chen, Aishan Liu, Xianglong Liu, Dacheng Tao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13563
• Pdf URL: https://arxiv.org/pdf/2501.13563
• Copy Paste: [[2501.13563]] Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving(https://arxiv.org/abs/2501.13563)
• Keywords: attack
• Abstract: Vision-language models (VLMs) have significantly advanced autonomous driving (AD) by enhancing reasoning capabilities; however, these models remain highly susceptible to adversarial attacks. While existing research has explored white-box attacks to some extent, the more practical and challenging black-box scenarios remain largely underexplored due to their inherent difficulty. In this paper, we take the first step toward designing black-box adversarial attacks specifically targeting VLMs in AD. We identify two key challenges for achieving effective black-box attacks in this context: the effectiveness across driving reasoning chains in AD systems and the dynamic nature of driving scenarios. To address this, we propose Cascading Adversarial Disruption (CAD). It first introduces Decision Chain Disruption, which targets low-level reasoning breakdown by generating and injecting deceptive semantics, ensuring the perturbations remain effective across the entire decision-making chain. Building on this, we present Risky Scene Induction, which addresses dynamic adaptation by leveraging a surrogate VLM to understand and construct high-level risky scenarios that are likely to result in critical errors in the current driving contexts. Extensive experiments conducted on multiple AD VLMs and benchmarks demonstrate that CAD achieves state-of-the-art attack effectiveness, significantly outperforming existing methods (+13.43% on average). Moreover, we validate its practical applicability through real-world attacks on AD vehicles powered by VLMs, where the route completion rate drops by 61.11% and the vehicle crashes directly into the obstacle vehicle with adversarial patches. Finally, we release CADA dataset, comprising 18,808 adversarial visual-question-answer pairs, to facilitate further evaluation and research in this critical domain. Our codes and dataset will be available after paper's acceptance.

Title: Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization

• Authors: Lei Huang, Xiaocheng Feng, Weitao Ma, Yuchun Fan, Xiachong Feng, Yangfan Ye, Weihong Zhong, Yuxuan Gu, Baoxin Wang, Dayong Wu, Guoping Hu, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13573
• Pdf URL: https://arxiv.org/pdf/2501.13573
• Copy Paste: [[2501.13573]] Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization(https://arxiv.org/abs/2501.13573)
• Keywords: large language model
• Abstract: Ensuring contextual faithfulness in retrieval-augmented large language models (LLMs) is crucial for building trustworthy information-seeking systems, particularly in long-form question-answering (LFQA) scenarios. In this work, we identify a salient correlation between LFQA faithfulness and retrieval heads, a set of attention heads responsible for retrieving contextual information. Leveraging this insight, we propose RHIO, a framework designed to teach LLMs to explicitly discriminate between faithful and unfaithful generations. RHIO first augments unfaithful samples that simulate realistic model-intrinsic errors by selectively masking retrieval heads. Then, these samples are incorporated into joint training, enabling the model to distinguish unfaithful outputs from faithful ones conditioned on control tokens. Furthermore, these control tokens are leveraged to self-induce contrastive outputs, amplifying their difference through contrastive decoding. Additionally, to facilitate the evaluation of contextual faithfulness, we also introduce GroundBench, a comprehensive benchmark compiled from five existing LFQA datasets. Extensive experimental results on GroundBench demonstrate that RHIO significantly improves faithfulness, even outperforming GPT-4o.

Title: Towards Robust Incremental Learning under Ambiguous Supervision

• Authors: Rui Wang, Mingxuan Xia, Chang Yao, Lei Feng, Junbo Zhao, Gang Chen, Haobo Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13584
• Pdf URL: https://arxiv.org/pdf/2501.13584
• Copy Paste: [[2501.13584]] Towards Robust Incremental Learning under Ambiguous Supervision(https://arxiv.org/abs/2501.13584)
• Keywords: robust
• Abstract: Traditional Incremental Learning (IL) targets to handle sequential fully-supervised learning problems where novel classes emerge from time to time. However, due to inherent annotation uncertainty and ambiguity, collecting high-quality annotated data in a dynamic learning system can be extremely expensive. To mitigate this problem, we propose a novel weakly-supervised learning paradigm called Incremental Partial Label Learning (IPLL), where the sequentially arrived data relate to a set of candidate labels rather than the ground truth. Technically, we develop the Prototype-Guided Disambiguation and Replay Algorithm (PGDR) which leverages the class prototypes as a proxy to mitigate two intertwined challenges in IPLL, i.e., label ambiguity and catastrophic forgetting. To handle the former, PGDR encapsulates a momentum-based pseudo-labeling algorithm along with prototype-guided initialization, resulting in a balanced perception of classes. To alleviate forgetting, we develop a memory replay technique that collects well-disambiguated samples while maintaining representativeness and diversity. By jointly distilling knowledge from curated memory data, our framework exhibits a great disambiguation ability for samples of new tasks and achieves less forgetting of knowledge. Extensive experiments demonstrate that PGDR achieves superior

Title: A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classification

• Authors: Younes Yousef, Lukas Galke, Ansgar Scherp
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13598
• Pdf URL: https://arxiv.org/pdf/2501.13598
• Copy Paste: [[2501.13598]] A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classification(https://arxiv.org/abs/2501.13598)
• Keywords: transformer
• Abstract: Recent approaches in hierarchical text classification (HTC) rely on the capabilities of a pre-trained transformer model and exploit the label semantics and a graph encoder for the label hierarchy. In this paper, we introduce an effective hierarchical text classifier RADAr (Transformer-based Autoregressive Decoder Architecture) that is based only on an off-the-shelf RoBERTa transformer to process the input and a custom autoregressive decoder with two decoder layers for generating the classification output. Thus, unlike existing approaches for HTC, the encoder of RADAr has no explicit encoding of the label hierarchy and the decoder solely relies on the label sequences of the samples observed during training. We demonstrate on three benchmark datasets that RADAr achieves results competitive to the state of the art with less training and inference time. Our model consistently performs better when organizing the label sequences from children to parents versus the inverse, as done in existing HTC approaches. Our experiments show that neither the label semantics nor an explicit graph encoder for the hierarchy is needed. This has strong practical implications for HTC as the architecture has fewer requirements and provides a speed-up by a factor of 2 at inference time. Moreover, training a separate decoder from scratch in conjunction with fine-tuning the encoder allows future researchers and practitioners to exchange the encoder part as new models arise. The source code is available at this https URL.

Title: FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences

• Authors: Maria Hartmann, Grégoire Danoy, Pascal Bouvry
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.13604
• Pdf URL: https://arxiv.org/pdf/2501.13604
• Copy Paste: [[2501.13604]] FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences(https://arxiv.org/abs/2501.13604)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) is a distributed machine learning strategy, developed for settings where training data is owned by distributed devices and cannot be shared. FL circumvents this constraint by carrying out model training in distribution. The parameters of these local models are shared intermittently among participants and aggregated to enhance model accuracy. This strategy has been rapidly adopted by the industry in efforts to overcome privacy and resource constraints in model training. However, the application of FL to real-world settings brings additional challenges associated with heterogeneity between participants. Research into mitigating these difficulties in FL has largely focused on only two types of heterogeneity: the unbalanced distribution of training data, and differences in client resources. Yet more types of heterogeneity are becoming relevant as the capability of FL expands to cover more complex problems, from the tuning of LLMs to enabling machine learning on edge devices. In this work, we discuss a novel type of heterogeneity that is likely to become increasingly relevant in future applications: this is preference heterogeneity, emerging when clients learn under multiple objectives, with different importance assigned to each objective on different clients. In this work, we discuss the implications of this type of heterogeneity and propose FedPref, a first algorithm designed to facilitate personalised FL in this setting. We demonstrate the effectiveness of the algorithm across different problems, preference distributions and model architectures. In addition, we introduce a new analytical point of view, based on multi-objective metrics, for evaluating the performance of FL algorithms in this setting beyond the traditional client-focused metrics. We perform a second experimental analysis based in this view, and show that FedPref outperforms compared algorithms.

Title: Cognitive Paradigms for Evaluating VLMs on Visual Reasoning Task

• Authors: Mohit Vaishnav, Tanel Tammet
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13620
• Pdf URL: https://arxiv.org/pdf/2501.13620
• Copy Paste: [[2501.13620]] Cognitive Paradigms for Evaluating VLMs on Visual Reasoning Task(https://arxiv.org/abs/2501.13620)
• Keywords: robust
• Abstract: Evaluating the reasoning capabilities of Vision-Language Models (VLMs) in complex visual tasks provides valuable insights into their potential and limitations. In this work, we assess the performance of VLMs on the challenging Bongard Openworld Problems benchmark, which involves reasoning over natural images. We propose and evaluate three human-inspired paradigms: holistic analysis (global context processing), deductive rule learning (explicit rule derivation and application), and componential analysis (structured decomposition of images into components). Our results demonstrate that state-of-the-art models, including GPT-4o and Gemini, not only surpass human benchmarks but also excel in structured reasoning tasks, with componential analysis proving especially effective. However, ablation studies reveal key challenges, such as handling synthetic images, making fine-grained distinctions, and interpreting nuanced contextual information. These insights underscore the need for further advancements in model robustness and generalization, while highlighting the transformative potential of structured reasoning approaches in enhancing VLM capabilities.

Title: Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models

• Authors: Zhenghao Lin, Zihao Tang, Xiao Liu, Yeyun Gong, Yi Cheng, Qi Chen, Hang Li, Ying Xin, Ziyue Yang, Kailai Yang, Yu Yan, Xiao Liang, Shuai Lu, Yiming Huang, Zheheng Luo, Lei Qu, Xuan Feng, Yaoxiang Wang, Yuqing Xia, Feiyang Chen, Yuting Jiang, Yasen Hu, Hao Ni, Binyang Li, Guoshuai Zhao, Jui-Hao Chiang, Zhongxin Guo, Chen Lin, Kun Kuang, Wenjie Li, Yelong Shen, Jian Jiao, Peng Cheng, Mao Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13629
• Pdf URL: https://arxiv.org/pdf/2501.13629
• Copy Paste: [[2501.13629]] Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models(https://arxiv.org/abs/2501.13629)
• Keywords: large language model
• Abstract: We introduce Sigma, an efficient large language model specialized for the system domain, empowered by a novel architecture including DiffQKV attention, and pre-trained on our meticulously collected system domain data. DiffQKV attention significantly enhances the inference efficiency of Sigma by optimizing the Query (Q), Key (K), and Value (V) components in the attention mechanism differentially, based on their varying impacts on the model performance and efficiency indicators. Specifically, we (1) conduct extensive experiments that demonstrate the model's varying sensitivity to the compression of K and V components, leading to the development of differentially compressed KV, and (2) propose augmented Q to expand the Q head dimension, which enhances the model's representation capacity with minimal impacts on the inference speed. Rigorous theoretical and empirical analyses reveal that DiffQKV attention significantly enhances efficiency, achieving up to a 33.36% improvement in inference speed over the conventional grouped-query attention (GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various sources, including 19.5B system domain data that we carefully collect and 1T tokens of synthesized and rewritten data. In general domains, Sigma achieves comparable performance to other state-of-arts models. In the system domain, we introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates remarkable performance across all tasks, significantly outperforming GPT-4 with an absolute improvement up to 52.5%.

Title: LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models

• Authors: Yizheng Sun, Yanze Xin, Hao Li, Jingyuan Sun, Chenghua Lin, Riza Batista-Navarro
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13652
• Pdf URL: https://arxiv.org/pdf/2501.13652
• Copy Paste: [[2501.13652]] LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models(https://arxiv.org/abs/2501.13652)
• Keywords: large language model
• Abstract: Multi-modal Large Language Models (MLLMs) have achieved remarkable success by integrating visual and textual modalities. However, they incur significant computational overhead due to the large number of vision tokens processed, limiting their practicality in resource-constrained environments. We introduce Language-Guided Vision Token Pruning (LVPruning) for MLLMs, an effective yet simple method that significantly reduces the computational burden while preserving model performance. LVPruning employs cross-attention modules to compute the importance of vision tokens based on their interaction with language tokens, determining which to prune. Importantly, LVPruning can be integrated without modifying the original MLLM parameters, which makes LVPruning simple to apply or remove. Our experiments show that LVPruning can effectively reduce up to 90% of vision tokens by the middle layer of LLaVA-1.5, resulting in a 62.1% decrease in inference Tera Floating-Point Operations Per Second (TFLOPs), with an average performance loss of just 0.45% across nine multi-modal benchmarks.

Title: MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation

• Authors: Fu Rong, Meng Lan, Qian Zhang, Lefei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13667
• Pdf URL: https://arxiv.org/pdf/2501.13667
• Copy Paste: [[2501.13667]] MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation(https://arxiv.org/abs/2501.13667)
• Keywords: segmentation
• Abstract: Referring video object segmentation (RVOS) aims to segment objects in a video according to textual descriptions, which requires the integration of multimodal information and temporal dynamics perception. The Segment Anything Model 2 (SAM 2) has shown great effectiveness across various video segmentation tasks. However, its application to offline RVOS is challenged by the translation of the text into effective prompts and a lack of global context awareness. In this paper, we propose a novel RVOS framework, termed MPG-SAM 2, to address these challenges. Specifically, MPG-SAM 2 employs a unified multimodal encoder to jointly encode video and textual features, generating semantically aligned video and text embeddings, along with multimodal class tokens. A mask prior generator utilizes the video embeddings and class tokens to create pseudo masks of target objects and global context. These masks are fed into the prompt encoder as dense prompts along with multimodal class tokens as sparse prompts to generate accurate prompts for SAM 2. To provide the online SAM 2 with a global view, we introduce a hierarchical global-historical aggregator, which allows SAM 2 to aggregate global and historical information of target objects at both pixel and object levels, enhancing the target representation and temporal consistency. Extensive experiments on several RVOS benchmarks demonstrate the superiority of MPG-SAM 2 and the effectiveness of our proposed modules.

Title: How to Complete Domain Tuning while Keeping General Ability in LLM: Adaptive Layer-wise and Element-wise Regularization

• Authors: Shezheng Song, Hao Xu, Jun Ma, Shasha Li, Long Peng, Qian Wan, Xiaodong Liu, Jie Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13669
• Pdf URL: https://arxiv.org/pdf/2501.13669
• Copy Paste: [[2501.13669]] How to Complete Domain Tuning while Keeping General Ability in LLM: Adaptive Layer-wise and Element-wise Regularization(https://arxiv.org/abs/2501.13669)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit strong general-purpose language capabilities. However, fine-tuning these models on domain-specific tasks often leads to catastrophic forgetting, where the model overwrites or loses essential knowledge acquired during pretraining. This phenomenon significantly limits the broader applicability of LLMs. To address this challenge, we propose a novel approach to compute the element-wise importance of model parameters crucial for preserving general knowledge during fine-tuning. Our method utilizes a dual-objective optimization strategy: (1) regularization loss to retain the parameter crucial for general knowledge; (2) cross-entropy loss to adapt to domain-specific tasks. Additionally, we introduce layer-wise coefficients to account for the varying contributions of different layers, dynamically balancing the dual-objective optimization. Extensive experiments on scientific, medical, and physical tasks using GPT-J and LLaMA-3 demonstrate that our approach mitigates catastrophic forgetting while enhancing model adaptability. Compared to previous methods, our solution is approximately 20 times faster and requires only 10%-15% of the storage, highlighting the practical efficiency. The code will be released.

Title: Certified Robustness Under Bounded Levenshtein Distance

• Authors: Elias Abad Rocamora, Grigorios G. Chrysos, Volkan Cevher
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13676
• Pdf URL: https://arxiv.org/pdf/2501.13676
• Copy Paste: [[2501.13676]] Certified Robustness Under Bounded Levenshtein Distance(https://arxiv.org/abs/2501.13676)
• Keywords: robust
• Abstract: Text classifiers suffer from small perturbations, that if chosen adversarially, can dramatically change the output of the model. Verification methods can provide robustness certificates against such adversarial perturbations, by computing a sound lower bound on the robust accuracy. Nevertheless, existing verification methods incur in prohibitive costs and cannot practically handle Levenshtein distance constraints. We propose the first method for computing the Lipschitz constant of convolutional classifiers with respect to the Levenshtein distance. We use these Lipschitz constant estimates for training 1-Lipschitz classifiers. This enables computing the certified radius of a classifier in a single forward pass. Our method, LipsLev, is able to obtain $38.80$% and $13.93$% verified accuracy at distance $1$ and $2$ respectively in the AG-News dataset, while being $4$ orders of magnitude faster than existing approaches. We believe our work can open the door to more efficient verification in the text domain.

Title: HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor

• Authors: Zihui Wu, Haichang Gao, Jiacheng Luo, Zhaoxiang Liu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13677
• Pdf URL: https://arxiv.org/pdf/2501.13677
• Copy Paste: [[2501.13677]] HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor(https://arxiv.org/abs/2501.13677)
• Keywords: defense, attack, robust, large language model
• Abstract: Large Language Models (LLMs) commonly rely on explicit refusal prefixes for safety, making them vulnerable to prefix injection attacks. We introduce HumorReject, a novel data-driven approach that fundamentally reimagines LLM safety by decoupling it from refusal prefixes through the use of humor as an indirect refusal strategy. Rather than explicitly rejecting harmful instructions, HumorReject responds with contextually appropriate humor that naturally defuses potentially dangerous requests while maintaining engaging interactions. Our approach effectively addresses the common "over-defense" issues in existing safety mechanisms, demonstrating superior robustness against various attack vectors while preserving natural and high-quality interactions on legitimate tasks. Our findings suggest that innovations at the data level are even more fundamental than the alignment algorithm itself in achieving effective LLM safety, opening new directions for developing more resilient and user-friendly AI systems.

Title: Unlearning Clients, Features and Samples in Vertical Federated Learning

• Authors: Ayush K. Varshney, Konstantinos Vandikas, Vicenç Torra
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13683
• Pdf URL: https://arxiv.org/pdf/2501.13683
• Copy Paste: [[2501.13683]] Unlearning Clients, Features and Samples in Vertical Federated Learning(https://arxiv.org/abs/2501.13683)
• Keywords: security, privacy, attack, membership infer, federate
• Abstract: Federated Learning (FL) has emerged as a prominent distributed learning paradigm. Within the scope of privacy preservation, information privacy regulations such as GDPR entitle users to request the removal (or unlearning) of their contribution from a service that is hosting the model. For this purpose, a server hosting an ML model must be able to unlearn certain information in cases such as copyright infringement or security issues that can make the model vulnerable or impact the performance of a service based on that model. While most unlearning approaches in FL focus on Horizontal FL (HFL), where clients share the feature space and the global model, Vertical FL (VFL) has received less attention from the research community. VFL involves clients (passive parties) sharing the sample space among them while not having access to the labels. In this paper, we explore unlearning in VFL from three perspectives: unlearning clients, unlearning features, and unlearning samples. To unlearn clients and features we introduce VFU-KD which is based on knowledge distillation (KD) while to unlearn samples, VFU-GA is introduced which is based on gradient ascent. To provide evidence of approximate unlearning, we utilize Membership Inference Attack (MIA) to audit the effectiveness of our unlearning approach. Our experiments across six tabular datasets and two image datasets demonstrate that VFU-KD and VFU-GA achieve performance comparable to or better than both retraining from scratch and the benchmark R2S method in many cases, with improvements of $(0-2\%)$. In the remaining cases, utility scores remain comparable, with a modest utility loss ranging from $1-5\%$. Unlike existing methods, VFU-KD and VFU-GA require no communication between active and passive parties during unlearning. However, they do require the active party to store the previously communicated embeddings.

Title: Question Answering on Patient Medical Records with Private Fine-Tuned LLMs

• Authors: Sara Kothari, Ayush Gupta
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13687
• Pdf URL: https://arxiv.org/pdf/2501.13687
• Copy Paste: [[2501.13687]] Question Answering on Patient Medical Records with Private Fine-Tuned LLMs(https://arxiv.org/abs/2501.13687)
• Keywords: privacy, large language model
• Abstract: Healthcare systems continuously generate vast amounts of electronic health records (EHRs), commonly stored in the Fast Healthcare Interoperability Resources (FHIR) standard. Despite the wealth of information in these records, their complexity and volume make it difficult for users to retrieve and interpret crucial health insights. Recent advances in Large Language Models (LLMs) offer a solution, enabling semantic question answering (QA) over medical data, allowing users to interact with their health records more effectively. However, ensuring privacy and compliance requires edge and private deployments of LLMs. This paper proposes a novel approach to semantic QA over EHRs by first identifying the most relevant FHIR resources for a user query (Task1) and subsequently answering the query based on these resources (Task2). We explore the performance of privately hosted, fine-tuned LLMs, evaluating them against benchmark models such as GPT-4 and GPT-4o. Our results demonstrate that fine-tuned LLMs, while 250x smaller in size, outperform GPT-4 family models by 0.55% in F1 score on Task1 and 42% on Meteor Task in Task2. Additionally, we examine advanced aspects of LLM usage, including sequential fine-tuning, model self-evaluation (narcissistic evaluation), and the impact of training data size on performance. The models and datasets are available here: this https URL

Title: Training-Free Consistency Pipeline for Fashion Repose

• Authors: Potito Aghilar, Vito Walter Anelli, Michelantonio Trizio, Tommaso Di Noia
• Subjects: cs.CV, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.13692
• Pdf URL: https://arxiv.org/pdf/2501.13692
• Copy Paste: [[2501.13692]] Training-Free Consistency Pipeline for Fashion Repose(https://arxiv.org/abs/2501.13692)
• Keywords: diffusion
• Abstract: Recent advancements in diffusion models have significantly broadened the possibilities for editing images of real-world objects. However, performing non-rigid transformations, such as changing the pose of objects or image-based conditioning, remains challenging. Maintaining object identity during these edits is difficult, and current methods often fall short of the precision needed for industrial applications, where consistency is critical. Additionally, fine-tuning diffusion models requires custom training data, which is not always accessible in real-world scenarios. This work introduces FashionRepose, a training-free pipeline for non-rigid pose editing specifically designed for the fashion industry. The approach integrates off-the-shelf models to adjust poses of long-sleeve garments, maintaining identity and branding attributes. FashionRepose uses a zero-shot approach to perform these edits in near real-time, eliminating the need for specialized training. consistent image editing. The solution holds potential for applications in the fashion industry and other fields demanding identity preservation in image editing.

Title: DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

• Authors: Linghao Zhang, Junhao Wang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Jiaheng Wen, Chengxing Xie, Maoquan Wang, Yufan Huang, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.13699
• Pdf URL: https://arxiv.org/pdf/2501.13699
• Copy Paste: [[2501.13699]] DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale(https://arxiv.org/abs/2501.13699)
• Keywords: robust, large language model
• Abstract: Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40\% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 42.9% execution pass rate, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis.

Title: A real-time battle situation intelligent awareness system based on Meta-learning & RNN

• Authors: Yuchun Li, Zihan Lin, Xize Wang, Chunyang Liu, Liaoyuan Wu, Fang Zhang
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2501.13704
• Pdf URL: https://arxiv.org/pdf/2501.13704
• Copy Paste: [[2501.13704]] A real-time battle situation intelligent awareness system based on Meta-learning & RNN(https://arxiv.org/abs/2501.13704)
• Keywords: attack
• Abstract: In modern warfare, real-time and accurate battle situation analysis is crucial for making strategic and tactical decisions. The proposed real-time battle situation intelligent awareness system (BSIAS) aims at meta-learning analysis and stepwise RNN (recurrent neural network) modeling, where the former carries out the basic processing and analysis of battlefield data, which includes multi-steps such as data cleansing, data fusion, data mining and continuously updates, and the latter optimizes the battlefield modeling by stepwise capturing the temporal dependencies of data set. BSIAS can predict the possible movement from any side of the fence and attack routes by taking a simulated battle as an example, which can be an intelligent support platform for commanders to make scientific decisions during wartime. This work delivers the potential application of integrated BSIAS in the field of battlefield command & analysis engineering.

Title: EventVL: Understand Event Streams via Multimodal Large Language Model

• Authors: Pengteng Li, Yunfan Lu, Pinghao Song, Wuyang Li, Huizai Yao, Hui Xiong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13707
• Pdf URL: https://arxiv.org/pdf/2501.13707
• Copy Paste: [[2501.13707]] EventVL: Understand Event Streams via Multimodal Large Language Model(https://arxiv.org/abs/2501.13707)
• Keywords: generative, large language model
• Abstract: The event-based Vision-Language Model (VLM) recently has made good progress for practical vision tasks. However, most of these works just utilize CLIP for focusing on traditional perception tasks, which obstruct model understanding explicitly the sufficient semantics and context from event streams. To address the deficiency, we propose EventVL, the first generative event-based MLLM (Multimodal Large Language Model) framework for explicit semantic understanding. Specifically, to bridge the data gap for connecting different modalities semantics, we first annotate a large event-image/video-text dataset, containing almost 1.4 million high-quality pairs of data, which enables effective learning across various scenes, e.g., drive scene or human motion. After that, we design Event Spatiotemporal Representation to fully explore the comprehensive information by diversely aggregating and segmenting the event stream. To further promote a compact semantic space, Dynamic Semantic Alignment is introduced to improve and complete sparse semantic spaces of events. Extensive experiments show that our EventVL can significantly surpass existing MLLM baselines in event captioning and scene description generation tasks. We hope our research could contribute to the development of the event vision community.

Title: A Mutual Information Perspective on Multiple Latent Variable Generative Models for Positive View Generation

• Authors: Dario Serez, Marco Cristani, Alessio Del Bue, Vittorio Murino, Pietro Morerio
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13718
• Pdf URL: https://arxiv.org/pdf/2501.13718
• Copy Paste: [[2501.13718]] A Mutual Information Perspective on Multiple Latent Variable Generative Models for Positive View Generation(https://arxiv.org/abs/2501.13718)
• Keywords: generative
• Abstract: In image generation, Multiple Latent Variable Generative Models (MLVGMs) employ multiple latent variables to gradually shape the final images, from global characteristics to finer and local details (e.g., StyleGAN, NVAE), emerging as powerful tools for diverse applications. Yet their generative dynamics and latent variable utilization remain only empirically observed. In this work, we propose a novel framework to systematically quantify the impact of each latent variable in MLVGMs, using Mutual Information (MI) as a guiding metric. Our analysis reveals underutilized variables and can guide the use of MLVGMs in downstream applications. With this foundation, we introduce a method for generating synthetic data for Self-Supervised Contrastive Representation Learning (SSCRL). By leveraging the hierarchical and disentangled variables of MLVGMs, and guided by the previous analysis, we apply tailored latent perturbations to produce diverse views for SSCRL, without relying on real data altogether. Additionally, we introduce a Continuous Sampling (CS) strategy, where the generator dynamically creates new samples during SSCRL training, greatly increasing data variability. Our comprehensive experiments demonstrate the effectiveness of these contributions, showing that MLVGMs' generated views compete on par with or even surpass views generated from real data. This work establishes a principled approach to understanding and exploiting MLVGMs, advancing both generative modeling and self-supervised learning.

Title: Musical ethnocentrism in Large Language Models

• Authors: Anna Kruspe
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.13720
• Pdf URL: https://arxiv.org/pdf/2501.13720
• Copy Paste: [[2501.13720]] Musical ethnocentrism in Large Language Models(https://arxiv.org/abs/2501.13720)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) reflect the biases in their training data and, by extension, those of the people who created this training data. Detecting, analyzing, and mitigating such biases is becoming a focus of research. One type of bias that has been understudied so far are geocultural biases. Those can be caused by an imbalance in the representation of different geographic regions and cultures in the training data, but also by value judgments contained therein. In this paper, we make a first step towards analyzing musical biases in LLMs, particularly ChatGPT and Mixtral. We conduct two experiments. In the first, we prompt LLMs to provide lists of the "Top 100" musical contributors of various categories and analyze their countries of origin. In the second experiment, we ask the LLMs to numerically rate various aspects of the musical cultures of different countries. Our results indicate a strong preference of the LLMs for Western music cultures in both experiments.

Title: RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation

• Authors: Shi-Qi Yan, Zhen-Hua Ling
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13726
• Pdf URL: https://arxiv.org/pdf/2501.13726
• Copy Paste: [[2501.13726]] RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation(https://arxiv.org/abs/2501.13726)
• Keywords: robust, large language model
• Abstract: While Retrieval-Augmented Generation (RAG) has exhibited promise in utilizing external knowledge, its generation process heavily depends on the quality and accuracy of the retrieved context. Large language models (LLMs) struggle to evaluate the correctness of non-parametric knowledge retrieved externally when it differs from internal memorization, leading to knowledge conflicts during response generation. To this end, we introduce the Retrieval Preference Optimization (RPO), a lightweight and effective alignment method to adaptively leverage multi-source knowledge based on retrieval relevance. An implicit representation of retrieval relevance is derived and incorporated into the reward model to integrate retrieval evaluation and response generation into a single model, solving the problem that previous methods necessitate the additional procedure to assess the retrieval quality. Notably, RPO is the only RAG-dedicated alignment approach that quantifies the awareness of retrieval relevance in training, overcoming mathematical obstacles. Experiments on four datasets demonstrate that RPO outperforms RAG by 4-10% in accuracy without any extra component, exhibiting its robust generalization.

Title: Pseudocode-Injection Magic: Enabling LLMs to Tackle Graph Computational Tasks

• Authors: Chang Gong, Wanrui Bian, Zhijie Zhang, Weiguo Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13731
• Pdf URL: https://arxiv.org/pdf/2501.13731
• Copy Paste: [[2501.13731]] Pseudocode-Injection Magic: Enabling LLMs to Tackle Graph Computational Tasks(https://arxiv.org/abs/2501.13731)
• Keywords: large language model
• Abstract: Graph computational tasks are inherently challenging and often demand the development of advanced algorithms for effective solutions. With the emergence of large language models (LLMs), researchers have begun investigating their potential to address these tasks. However, existing approaches are constrained by LLMs' limited capability to comprehend complex graph structures and their high inference costs, rendering them impractical for handling large-scale graphs. Inspired by human approaches to graph problems, we introduce a novel framework, PIE (Pseudocode-Injection-Enhanced LLM Reasoning for Graph Computational Tasks), which consists of three key steps: problem understanding, prompt design, and code generation. In this framework, LLMs are tasked with understanding the problem and extracting relevant information to generate correct code. The responsibility for analyzing the graph structure and executing the code is delegated to the interpreter. We inject task-related pseudocodes into the prompts to further assist the LLMs in generating efficient code. We also employ cost-effective trial-and-error techniques to ensure that the LLM-generated code executes correctly. Unlike other methods that require invoking LLMs for each individual test case, PIE only calls the LLM during the code generation phase, allowing the generated code to be reused and significantly reducing inference costs. Extensive experiments demonstrate that PIE outperforms existing baselines in terms of both accuracy and computational efficiency.

Title: Post-Quantum Stealth Address Protocols

• Authors: Marija Mikic, Mihajlo Srbakoski, Strahinja Praska
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13733
• Pdf URL: https://arxiv.org/pdf/2501.13733
• Copy Paste: [[2501.13733]] Post-Quantum Stealth Address Protocols(https://arxiv.org/abs/2501.13733)
• Keywords: privacy, attack, steal
• Abstract: The Stealth Address Protocol (SAP) allows users to receive assets through stealth addresses that are unlinkable to their stealth meta-addresses. The most widely used SAP, Dual-Key SAP (DKSAP), and the most performant SAP, Elliptic Curve Pairing Dual-Key SAP (ECPDKSAP), are based on elliptic curve cryptography, which is vulnerable to quantum attacks. These protocols depend on the elliptic curve discrete logarithm problem, which could be efficiently solved on a sufficiently powerful quantum computer using the Shor algorithm. In this paper three novel post-quantum SAPs based on lattice-based cryptography are presented: LWE SAP, Ring-LWE SAP and Module-LWE SAP. These protocols leverage Learning With Errors (LWE) problem to ensure quantum-resistant privacy. Among them, Module-LWE SAP, which is based on the Kyber key encapsulation mechanism, achieves the best performance and outperforms ECPDKSAP by approximately 66.8% in the scan time of the ephemeral public key registry.

Title: GPT-HTree: A Decision Tree Framework Integrating Hierarchical Clustering and Large Language Models for Explainable Classification

• Authors: Te Pei, Fuat Alican, Aaron Ontoyin Yin, Yigit Ihlamur
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13743
• Pdf URL: https://arxiv.org/pdf/2501.13743
• Copy Paste: [[2501.13743]] GPT-HTree: A Decision Tree Framework Integrating Hierarchical Clustering and Large Language Models for Explainable Classification(https://arxiv.org/abs/2501.13743)
• Keywords: interpretability, large language model
• Abstract: This paper introduces GPT-HTree, a framework combining hierarchical clustering, decision trees, and large language models (LLMs) to address this challenge. By leveraging hierarchical clustering to segment individuals based on salient features, resampling techniques to balance class distributions, and decision trees to tailor classification paths within each cluster, GPT-HTree ensures both accuracy and interpretability. LLMs enhance the framework by generating human-readable cluster descriptions, bridging quantitative analysis with actionable insights.

Title: Exact Soft Analytical Side-Channel Attacks using Tractable Circuits

• Authors: Thomas Wedenig, Rishub Nagpal, Gaëtan Cassiers, Stefan Mangard, Robert Peharz
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13748
• Pdf URL: https://arxiv.org/pdf/2501.13748
• Copy Paste: [[2501.13748]] Exact Soft Analytical Side-Channel Attacks using Tractable Circuits(https://arxiv.org/abs/2501.13748)
• Keywords: secure, attack
• Abstract: Detecting weaknesses in cryptographic algorithms is of utmost importance for designing secure information systems. The state-of-the-art soft analytical side-channel attack (SASCA) uses physical leakage information to make probabilistic predictions about intermediate computations and combines these "guesses" with the known algorithmic logic to compute the posterior distribution over the key. This attack is commonly performed via loopy belief propagation, which, however, lacks guarantees in terms of convergence and inference quality. In this paper, we develop a fast and exact inference method for SASCA, denoted as ExSASCA, by leveraging knowledge compilation and tractable probabilistic circuits. When attacking the Advanced Encryption Standard (AES), the most widely used encryption algorithm to date, ExSASCA outperforms SASCA by more than 31% top-1 success rate absolute. By leveraging sparse belief messages, this performance is achieved with little more computational cost than SASCA, and about 3 orders of magnitude less than exact inference via exhaustive enumeration. Even with dense belief messages, ExSASCA still uses 6 times less computations than exhaustive inference.

Title: 2-Tier SimCSE: Elevating BERT for Robust Sentence Embeddings

• Authors: Yumeng Wang, Ziran Zhou, Junjin Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13758
• Pdf URL: https://arxiv.org/pdf/2501.13758
• Copy Paste: [[2501.13758]] 2-Tier SimCSE: Elevating BERT for Robust Sentence Embeddings(https://arxiv.org/abs/2501.13758)
• Keywords: robust
• Abstract: Effective sentence embeddings that capture semantic nuances and generalize well across diverse contexts are crucial for natural language processing tasks. We address this challenge by applying SimCSE (Simple Contrastive Learning of Sentence Embeddings) using contrastive learning to fine-tune the minBERT model for sentiment analysis, semantic textual similarity (STS), and paraphrase detection. Our contributions include experimenting with three different dropout techniques, namely standard dropout, curriculum dropout, and adaptive dropout, to tackle overfitting, proposing a novel 2-Tier SimCSE Fine-tuning Model that combines both unsupervised and supervised SimCSE on STS task, and exploring transfer learning potential for Paraphrase and SST tasks. Our findings demonstrate the effectiveness of SimCSE, with the 2-Tier model achieving superior performance on the STS task, with an average test score of 0.742 across all three downstream tasks. The results of error analysis reveals challenges in handling complex sentiments and reliance on lexical overlap for paraphrase detection, highlighting areas for future research. The ablation study revealed that removing Adaptive Dropout in the Single-Task Unsupervised SimCSE Model led to improved performance on the STS task, indicating overfitting due to added parameters. Transfer learning from SimCSE models on Paraphrase and SST tasks did not enhance performance, suggesting limited transferability of knowledge from the STS task.

Title: UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models

• Authors: Xin Xu, Jiaxin Zhang, Tianhao Chen, Zitong Chao, Jishan Hu, Can Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13766
• Pdf URL: https://arxiv.org/pdf/2501.13766
• Copy Paste: [[2501.13766]] UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models(https://arxiv.org/abs/2501.13766)
• Keywords: robust, fair, large language model
• Abstract: Large Language Models (LLMs) have made significant strides in mathematical reasoning, underscoring the need for a comprehensive and fair evaluation of their capabilities. However, existing benchmarks often fall short, either lacking extensive coverage of undergraduate-level mathematical problems or probably suffering from test-set contamination. To address these issues, we introduce UGMathBench, a diverse and dynamic benchmark specifically designed for evaluating undergraduate-level mathematical reasoning with LLMs. UGMathBench comprises 5,062 problems across 16 subjects and 111 topics, featuring 10 distinct answer types. Each problem includes three randomized versions, with additional versions planned for release as leading open-source LLMs become saturated in UGMathBench. Furthermore, we propose two key metrics: effective accuracy (EAcc), which measures the percentage of correctly solved problems across all three versions, and reasoning gap ($\Delta$), which assesses reasoning robustness by calculating the difference between the average accuracy across all versions and EAcc. Our extensive evaluation of 23 leading LLMs reveals that the highest EAcc achieved is 56.3\% by OpenAI-o1-mini, with large $\Delta$ values observed across different models. This highlights the need for future research aimed at developing "large reasoning models" with high EAcc and $\Delta = 0$. We anticipate that the release of UGMathBench, along with its detailed evaluation codes, will serve as a valuable resource to advance the development of LLMs in solving mathematical problems.

Title: An Efficient Diffusion-based Non-Autoregressive Solver for Traveling Salesman Problem

• Authors: Mingzhao Wang, You Zhou, Zhiguang Cao, Yubin Xiao, Xuan Wu, Wei Pang, Yuan Jiang, Hui Yang, Peng Zhao, Yuanshu Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13767
• Pdf URL: https://arxiv.org/pdf/2501.13767
• Copy Paste: [[2501.13767]] An Efficient Diffusion-based Non-Autoregressive Solver for Traveling Salesman Problem(https://arxiv.org/abs/2501.13767)
• Keywords: extraction, diffusion, transformer
• Abstract: Recent advances in neural models have shown considerable promise in solving Traveling Salesman Problems (TSPs) without relying on much hand-crafted engineering. However, while non-autoregressive (NAR) approaches benefit from faster inference through parallelism, they typically deliver solutions of inferior quality compared to autoregressive ones. To enhance the solution quality while maintaining fast inference, we propose DEITSP, a diffusion model with efficient iterations tailored for TSP that operates in a NAR manner. Firstly, we introduce a one-step diffusion model that integrates the controlled discrete noise addition process with self-consistency enhancement, enabling optimal solution prediction through simultaneous denoising of multiple solutions. Secondly, we design a dual-modality graph transformer to bolster the extraction and fusion of features from node and edge modalities, while further accelerating the inference with fewer layers. Thirdly, we develop an efficient iterative strategy that alternates between adding and removing noise to improve exploration compared to previous diffusion methods. Additionally, we devise a scheduling framework to progressively refine the solution space by adjusting noise levels, facilitating a smooth search for optimal solutions. Extensive experiments on real-world and large-scale TSP instances demonstrate that DEITSP performs favorably against existing neural approaches in terms of solution quality, inference latency, and generalization ability. Our code is available at $\href{this https URL}{this https URL}$.

Title: Aggregating Digital Identities through Bridging. An Integration of Open Authentication Protocols for Web3 Identifiers

• Authors: Ben Biedermann, Matthew Scerri, Victoria Kozlova, Joshua Ellul
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.13770
• Pdf URL: https://arxiv.org/pdf/2501.13770
• Copy Paste: [[2501.13770]] Aggregating Digital Identities through Bridging. An Integration of Open Authentication Protocols for Web3 Identifiers(https://arxiv.org/abs/2501.13770)
• Keywords: privacy
• Abstract: Web3's decentralised infrastructure has upended the standardised approach to digital identity established by protocols like OpenID Connect. Web2 and Web3 currently operate in silos, with Web2 leveraging selective disclosure JSON web tokens (SD-JWTs) and Web3 dApps being reliant on on-chain data and sometimes clinging to centralised system data. This fragmentation hinders user experience and the interconnectedness of the digital world. This paper explores the integration of Web3 within the OpenID Connect framework, scrutinising established authentication protocols for their adaptability to decentralised identities. The research examines the interplay between OpenID Connect and decentralised identity concepts, the limitations of existing protocols like OpenID Connect for verifiable credential issuance, OpenID Connect framework for verifiable presentations, and self-issued OpenID provider. As a result, a novel privacy-preserving digital identity bridge is proposed, which aims to answer the research question of whether authentication protocols should inherently support Web3 functionalities and the mechanisms for their integration. Through a Decentralised Autonomous Organisation (DAO) use case, the findings indicate that a privacy-centric bridge can mitigate existing fragmentation by aggregating different identities to provide a better user experience. While the digital identity bridge demonstrates a possible approach to harmonise digital identity across platforms for their use in Web3, the bridging is unidirectional and limits root trust of credentials. The bridge's dependence on centralised systems may further fuel the debate on (de-)centralised identities.

Title: Do Large Language Models Truly Understand Geometric Structures?

• Authors: Xiaofeng Wang, Yiming Wang, Wenhong Zhu, Rui Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13773
• Pdf URL: https://arxiv.org/pdf/2501.13773
• Copy Paste: [[2501.13773]] Do Large Language Models Truly Understand Geometric Structures?(https://arxiv.org/abs/2501.13773)
• Keywords: large language model
• Abstract: Geometric ability is a significant challenge for large language models (LLMs) due to the need for advanced spatial comprehension and abstract thinking. Existing datasets primarily evaluate LLMs on their final answers, but they cannot truly measure their true understanding of geometric structures, as LLMs can arrive at correct answers by coincidence. To fill this gap, we introduce the GeomRel dataset, designed to evaluate LLMs' understanding of geometric structures by isolating the core step of geometric relationship identification in problem-solving. Using this benchmark, we conduct thorough evaluations of diverse LLMs and identify key limitations in understanding geometric structures. We further propose the Geometry Chain-of-Thought (GeoCoT) method, which enhances LLMs' ability to identify geometric relationships, resulting in significant performance improvements.

Title: Crossfire: An Elastic Defense Framework for Graph Neural Networks Under Bit Flip Attacks

• Authors: Lorenz Kummer, Samir Moustafa, Wilfried Gansterer, Nils Kriege
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13776
• Pdf URL: https://arxiv.org/pdf/2501.13776
• Copy Paste: [[2501.13776]] Crossfire: An Elastic Defense Framework for Graph Neural Networks Under Bit Flip Attacks(https://arxiv.org/abs/2501.13776)
• Keywords: defense, attack
• Abstract: Bit Flip Attacks (BFAs) are a well-established class of adversarial attacks, originally developed for Convolutional Neural Networks within the computer vision domain. Most recently, these attacks have been extended to target Graph Neural Networks (GNNs), revealing significant vulnerabilities. This new development naturally raises questions about the best strategies to defend GNNs against BFAs, a challenge for which no solutions currently exist. Given the applications of GNNs in critical fields, any defense mechanism must not only maintain network performance, but also verifiably restore the network to its pre-attack state. Verifiably restoring the network to its pre-attack state also eliminates the need for costly evaluations on test data to ensure network quality. We offer first insights into the effectiveness of existing honeypot- and hashing-based defenses against BFAs adapted from the computer vision domain to GNNs, and characterize the shortcomings of these approaches. To overcome their limitations, we propose Crossfire, a hybrid approach that exploits weight sparsity and combines hashing and honeypots with bit-level correction of out-of-distribution weight elements to restore network integrity. Crossfire is retraining-free and does not require labeled data. Averaged over 2,160 experiments on six benchmark datasets, Crossfire offers a 21.8% higher probability than its competitors of reconstructing a GNN attacked by a BFA to its pre-attack state. These experiments cover up to 55 bit flips from various attacks. Moreover, it improves post-repair prediction quality by 10.85%. Computational and storage overheads are negligible compared to the inherent complexity of even the simplest GNNs.

Title: Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling

• Authors: Tanya Rodchenko, Natasha Noy, Nino Scherrer, Jennifer Prendki
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13779
• Pdf URL: https://arxiv.org/pdf/2501.13779
• Copy Paste: [[2501.13779]] Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling(https://arxiv.org/abs/2501.13779)
• Keywords: large language model
• Abstract: While Large Language Models require more and more data to train and scale, rather than looking for any data to acquire, we should consider what types of tasks are more likely to benefit from data scaling. We should be intentional in our data acquisition. We argue that the topology of data itself informs which tasks to prioritize in data scaling, and shapes the development of the next generation of compute paradigms for tasks where data scaling is inefficient, or even insufficient.

Title: Defending against Adversarial Malware Attacks on ML-based Android Malware Detection Systems

• Authors: Ping He, Lorenzo Cavallaro, Shouling Ji
• Subjects: cs.CR, cs.AI, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.13782
• Pdf URL: https://arxiv.org/pdf/2501.13782
• Copy Paste: [[2501.13782]] Defending against Adversarial Malware Attacks on ML-based Android Malware Detection Systems(https://arxiv.org/abs/2501.13782)
• Keywords: privacy, protect, defense, attack, robust
• Abstract: Android malware presents a persistent threat to users' privacy and data integrity. To combat this, researchers have proposed machine learning-based (ML-based) Android malware detection (AMD) systems. However, adversarial Android malware attacks compromise the detection integrity of the ML-based AMD systems, raising significant concerns. Existing defenses against adversarial Android malware provide protections against feature space attacks which generate adversarial feature vectors only, leaving protection against realistic threats from problem space attacks which generate real adversarial malware an open problem. In this paper, we address this gap by proposing ADD, a practical adversarial Android malware defense framework designed as a plug-in to enhance the adversarial robustness of the ML-based AMD systems against problem space attacks. Our extensive evaluation across various ML-based AMD systems demonstrates that ADD is effective against state-of-the-art problem space adversarial Android malware attacks. Additionally, ADD shows the defense effectiveness in enhancing the adversarial robustness of real-world antivirus solutions.

Title: Parameter-Efficient Fine-Tuning for Foundation Models

• Authors: Dan Zhang, Tao Feng, Lilong Xue, Yuandong Wang, Yuxiao Dong, Jie Tang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13787
• Pdf URL: https://arxiv.org/pdf/2501.13787
• Copy Paste: [[2501.13787]] Parameter-Efficient Fine-Tuning for Foundation Models(https://arxiv.org/abs/2501.13787)
• Keywords: generative
• Abstract: This survey delves into the realm of Parameter-Efficient Fine-Tuning (PEFT) within the context of Foundation Models (FMs). PEFT, a cost-effective fine-tuning technique, minimizes parameters and computational complexity while striving for optimal downstream task performance. FMs, like ChatGPT, DALL-E, and LLaVA specialize in language understanding, generative tasks, and multimodal tasks, trained on diverse datasets spanning text, images, and videos. The diversity of FMs guides various adaptation strategies for PEFT. Therefore, this survey aims to provide a comprehensive overview of PEFT techniques applied to diverse FMs and address critical gaps in understanding the techniques, trends, and applications. We start by providing a detailed development of FMs and PEFT. Subsequently, we systematically review the key categories and core mechanisms of PEFT across diverse FMs to offer a comprehensive understanding of trends. We also explore the most recent applications across various FMs to demonstrate the versatility of PEFT, shedding light on the integration of systematic PEFT methods with a range of FMs. Furthermore, we identify potential research and development directions for improving PEFTs in the future. This survey provides a valuable resource for both newcomers and experts seeking to understand and use the power of PEFT across FMs. All reviewed papers are listed at \url{this https URL}.

Title: Unveiling the Power of Noise Priors: Enhancing Diffusion Models for Mobile Traffic Prediction

• Authors: Zhi Sheng, Yuan Yuan, Jingtao Ding, Yong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13794
• Pdf URL: https://arxiv.org/pdf/2501.13794
• Copy Paste: [[2501.13794]] Unveiling the Power of Noise Priors: Enhancing Diffusion Models for Mobile Traffic Prediction(https://arxiv.org/abs/2501.13794)
• Keywords: robust, diffusion
• Abstract: Accurate prediction of mobile traffic, \textit{i.e.,} network traffic from cellular base stations, is crucial for optimizing network performance and supporting urban development. However, the non-stationary nature of mobile traffic, driven by human activity and environmental changes, leads to both regular patterns and abrupt variations. Diffusion models excel in capturing such complex temporal dynamics due to their ability to capture the inherent uncertainties. Most existing approaches prioritize designing novel denoising networks but often neglect the critical role of noise itself, potentially leading to sub-optimal performance. In this paper, we introduce a novel perspective by emphasizing the role of noise in the denoising process. Our analysis reveals that noise fundamentally shapes mobile traffic predictions, exhibiting distinct and consistent patterns. We propose NPDiff, a framework that decomposes noise into \textit{prior} and \textit{residual} components, with the \textit{prior} derived from data dynamics, enhancing the model's ability to capture both regular and abrupt variations. NPDiff can seamlessly integrate with various diffusion-based prediction models, delivering predictions that are effective, efficient, and robust. Extensive experiments demonstrate that it achieves superior performance with an improvement over 30\%, offering a new perspective on leveraging diffusion models in this domain.

Title: Rudraksh: A compact and lightweight post-quantum key-encapsulation mechanism

• Authors: Suparna Kundu, Archisman Ghosh, Angshuman Karmakar, Shreyas Sen, Ingrid Verbauwhede
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13799
• Pdf URL: https://arxiv.org/pdf/2501.13799
• Copy Paste: [[2501.13799]] Rudraksh: A compact and lightweight post-quantum key-encapsulation mechanism(https://arxiv.org/abs/2501.13799)
• Keywords: security, attack
• Abstract: Resource-constrained devices such as wireless sensors and Internet of Things (IoT) devices have become ubiquitous in our digital ecosystem. These devices generate and handle a major part of our digital data. However, due to the impending threat of quantum computers on our existing public-key cryptographic schemes and the limited resources available on IoT devices, it is important to design lightweight post-quantum cryptographic (PQC) schemes suitable for these devices. In this work, we explored the design space of learning with error-based PQC schemes to design a lightweight key-encapsulation mechanism (KEM) suitable for resource-constrained devices. We have done a scrupulous and extensive analysis and evaluation of different design elements, such as polynomial size, field modulus structure, reduction algorithm, and secret and error distribution of an LWE-based KEM. Our explorations led to the proposal of a lightweight PQC-KEM, Rudraksh, without compromising security. Our scheme provides security against chosen ciphertext attacks (CCA) with more than 100 bits of Core-SVP post-quantum security and belongs to the NIST-level-I security category (provide security at least as much as AES-128). We have also shown how ASCON can be used for lightweight pseudo-random number generation and hash function in the lattice-based KEMs instead of the widely used Keccak for lightweight design. Our FPGA results show that Rudraksh currently requires the least area among the PQC KEMs of similar security. Our implementation of Rudraksh provides a $\sim3\times$ improvement in terms of the area requirement compared to the state-of-the-art area-optimized implementation of Kyber, can operate at $63\%$-$76\%$ higher frequency with respect to high-throughput Kyber, and improves time-area-product $\sim2\times$ compared to the state-of-the-art compact implementation of Kyber published in HPEC 2022.

Title: EgoHand: Ego-centric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMUs

• Authors: Yizhe Lv, Tingting Zhang, Yunpeng Song, Han Ding, Jinsong Han, Fei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13805
• Pdf URL: https://arxiv.org/pdf/2501.13805
• Copy Paste: [[2501.13805]] EgoHand: Ego-centric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMUs(https://arxiv.org/abs/2501.13805)
• Keywords: privacy, protect, robust, transformer
• Abstract: Recent advanced Virtual Reality (VR) headsets, such as the Apple Vision Pro, employ bottom-facing cameras to detect hand gestures and inputs, which offers users significant convenience in VR interactions. However, these bottom-facing cameras can sometimes be inconvenient and pose a risk of unintentionally exposing sensitive information, such as private body parts or personal surroundings. To mitigate these issues, we introduce EgoHand. This system provides an alternative solution by integrating millimeter-wave radar and IMUs for hand gesture recognition, thereby offering users an additional option for gesture interaction that enhances privacy protection. To accurately recognize hand gestures, we devise a two-stage skeleton-based gesture recognition scheme. In the first stage, a novel end-to-end Transformer architecture is employed to estimate the coordinates of hand joints. Subsequently, these estimated joint coordinates are utilized for gesture recognition. Extensive experiments involving 10 subjects show that EgoHand can detect hand gestures with 90.8% accuracy. Furthermore, EgoHand demonstrates robust performance across a variety of cross-domain tests, including different users, dominant hands, body postures, and scenes.

Title: Hallucinations Can Improve Large Language Models in Drug Discovery

• Authors: Shuzhou Yuan, Michael Färber
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13824
• Pdf URL: https://arxiv.org/pdf/2501.13824
• Copy Paste: [[2501.13824]] Hallucinations Can Improve Large Language Models in Drug Discovery(https://arxiv.org/abs/2501.13824)
• Keywords: large language model
• Abstract: Concerns about hallucinations in Large Language Models (LLMs) have been raised by researchers, yet their potential in areas where creativity is vital, such as drug discovery, merits exploration. In this paper, we come up with the hypothesis that hallucinations can improve LLMs in drug discovery. To verify this hypothesis, we use LLMs to describe the SMILES string of molecules in natural language and then incorporate these descriptions as part of the prompt to address specific tasks in drug discovery. Evaluated on seven LLMs and five classification tasks, our findings confirm the hypothesis: LLMs can achieve better performance with text containing hallucinations. Notably, Llama-3.1-8B achieves an 18.35% gain in ROC-AUC compared to the baseline without hallucination. Furthermore, hallucinations generated by GPT-4o provide the most consistent improvements across models. Additionally, we conduct empirical analyses and a case study to investigate key factors affecting performance and the underlying reasons. Our research sheds light on the potential use of hallucinations for LLMs and offers new perspectives for future research leveraging LLMs in drug discovery.

Title: MV-GMN: State Space Model for Multi-View Action Recognition

• Authors: Yuhui Lin, Jiaxuan Lu, Yue Yong, Jiahao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13829
• Pdf URL: https://arxiv.org/pdf/2501.13829
• Copy Paste: [[2501.13829]] MV-GMN: State Space Model for Multi-View Action Recognition(https://arxiv.org/abs/2501.13829)
• Keywords: transformer
• Abstract: Recent advancements in multi-view action recognition have largely relied on Transformer-based models. While effective and adaptable, these models often require substantial computational resources, especially in scenarios with multiple views and multiple temporal sequences. Addressing this limitation, this paper introduces the MV-GMN model, a state-space model specifically designed to efficiently aggregate multi-modal data (RGB and skeleton), multi-view perspectives, and multi-temporal information for action recognition with reduced computational complexity. The MV-GMN model employs an innovative Multi-View Graph Mamba network comprising a series of MV-GMN blocks. Each block includes a proposed Bidirectional State Space Block and a GCN module. The Bidirectional State Space Block introduces four scanning strategies, including view-prioritized and time-prioritized approaches. The GCN module leverages rule-based and KNN-based methods to construct the graph network, effectively integrating features from different viewpoints and temporal instances. Demonstrating its efficacy, MV-GMN outperforms the state-of-the-arts on several datasets, achieving notable accuracies of 97.3\% and 96.7\% on the NTU RGB+D 120 dataset in cross-subject and cross-view scenarios, respectively. MV-GMN also surpasses Transformer-based baselines while requiring only linear inference complexity, underscoring the model's ability to reduce computational load and enhance the scalability and applicability of multi-view action recognition technologies.

Title: Predicting Compact Phrasal Rewrites with Large Language Models for ASR Post Editing

• Authors: Hao Zhang, Felix Stahlberg, Shankar Kumar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13831
• Pdf URL: https://arxiv.org/pdf/2501.13831
• Copy Paste: [[2501.13831]] Predicting Compact Phrasal Rewrites with Large Language Models for ASR Post Editing(https://arxiv.org/abs/2501.13831)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) excel at rewriting tasks such as text style transfer and grammatical error correction. While there is considerable overlap between the inputs and outputs in these tasks, the decoding cost still increases with output length, regardless of the amount of overlap. By leveraging the overlap between the input and the output, Kaneko and Okazaki (2023) proposed model-agnostic edit span representations to compress the rewrites to save computation. They reported an output length reduction rate of nearly 80% with minimal accuracy impact in four rewriting tasks. In this paper, we propose alternative edit phrase representations inspired by phrase-based statistical machine translation. We systematically compare our phrasal representations with their span representations. We apply the LLM rewriting model to the task of Automatic Speech Recognition (ASR) post editing and show that our target-phrase-only edit representation has the best efficiency-accuracy trade-off. On the LibriSpeech test set, our method closes 50-60% of the WER gap between the edit span model and the full rewrite model while losing only 10-20% of the length reduction rate of the edit span model.

Title: Where Do You Go? Pedestrian Trajectory Prediction using Scene Features

• Authors: Mohammad Ali Rezaei, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13848
• Pdf URL: https://arxiv.org/pdf/2501.13848
• Copy Paste: [[2501.13848]] Where Do You Go? Pedestrian Trajectory Prediction using Scene Features(https://arxiv.org/abs/2501.13848)
• Keywords: segmentation
• Abstract: Accurate prediction of pedestrian trajectories is crucial for enhancing the safety of autonomous vehicles and reducing traffic fatalities involving pedestrians. While numerous studies have focused on modeling interactions among pedestrians to forecast their movements, the influence of environmental factors and scene-object placements has been comparatively underexplored. In this paper, we present a novel trajectory prediction model that integrates both pedestrian interactions and environmental context to improve prediction accuracy. Our approach captures spatial and temporal interactions among pedestrians within a sparse graph framework. To account for pedestrian-scene interactions, we employ advanced image enhancement and semantic segmentation techniques to extract detailed scene features. These scene and interaction features are then fused through a cross-attention mechanism, enabling the model to prioritize relevant environmental factors that influence pedestrian movements. Finally, a temporal convolutional network processes the fused features to predict future pedestrian trajectories. Experimental results demonstrate that our method significantly outperforms existing state-of-the-art approaches, achieving ADE and FDE values of 0.252 and 0.372 meters, respectively, underscoring the importance of incorporating both social interactions and environmental context in pedestrian trajectory prediction.

Title: A RAG-Based Institutional Assistant

• Authors: Gustavo Kuratomi, Paulo Pirozelli, Fabio G. Cozman, Sarajane M. Peres
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13880
• Pdf URL: https://arxiv.org/pdf/2501.13880
• Copy Paste: [[2501.13880]] A RAG-Based Institutional Assistant(https://arxiv.org/abs/2501.13880)
• Keywords: generative, large language model
• Abstract: Although large language models (LLMs) demonstrate strong text generation capabilities, they struggle in scenarios requiring access to structured knowledge bases or specific documents, limiting their effectiveness in knowledge-intensive tasks. To address this limitation, retrieval-augmented generation (RAG) models have been developed, enabling generative models to incorporate relevant document fragments into their inputs. In this paper, we design and evaluate a RAG-based virtual assistant specifically tailored for the University of São Paulo. Our system architecture comprises two key modules: a retriever and a generative model. We experiment with different types of models for both components, adjusting hyperparameters such as chunk size and the number of retrieved documents. Our optimal retriever model achieves a Top-5 accuracy of 30%, while our most effective generative model scores 22.04\% against ground truth answers. Notably, when the correct document chunks are supplied to the LLMs, accuracy significantly improves to 54.02%, an increase of over 30 percentage points. Conversely, without contextual input, performance declines to 13.68%. These findings highlight the critical role of database access in enhancing LLM performance. They also reveal the limitations of current semantic search methods in accurately identifying relevant documents and underscore the ongoing challenges LLMs face in generating precise responses.

Title: Utilizing Evolution Strategies to Train Transformers in Reinforcement Learning

• Authors: Matyáš Lorenc
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2501.13883
• Pdf URL: https://arxiv.org/pdf/2501.13883
• Copy Paste: [[2501.13883]] Utilizing Evolution Strategies to Train Transformers in Reinforcement Learning(https://arxiv.org/abs/2501.13883)
• Keywords: transformer
• Abstract: We explore a capability of evolution strategies to train an agent with its policy based on a transformer architecture in a reinforcement learning setting. We performed experiments using OpenAI's highly parallelizable evolution strategy to train Decision Transformer in Humanoid locomotion environment and in the environment of Atari games, testing the ability of this black-box optimization technique to train even such relatively large and complicated models (compared to those previously tested in the literature). We also proposed a method to aid the training by first pretraining the model before using the OpenAI-ES to train it further, and tested its effectiveness. The examined evolution strategy proved to be, in general, capable of achieving strong results and managed to obtain high-performing agents. Therefore, the pretraining was shown to be unnecessary; yet still, it helped us observe and formulate several further insights.

Title: What Does an Audio Deepfake Detector Focus on? A Study in the Time Domain

• Authors: Petr Grinberg, Ankur Kumar, Surya Koppisetti, Gaurav Bharaj
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.13887
• Pdf URL: https://arxiv.org/pdf/2501.13887
• Copy Paste: [[2501.13887]] What Does an Audio Deepfake Detector Focus on? A Study in the Time Domain(https://arxiv.org/abs/2501.13887)
• Keywords: transformer
• Abstract: Adding explanations to audio deepfake detection (ADD) models will boost their real-world application by providing insight on the decision making process. In this paper, we propose a relevancy-based explainable AI (XAI) method to analyze the predictions of transformer-based ADD models. We compare against standard Grad-CAM and SHAP-based methods, using quantitative faithfulness metrics as well as a partial spoof test, to comprehensively analyze the relative importance of different temporal regions in an audio. We consider large datasets, unlike previous works where only limited utterances are studied, and find that the XAI methods differ in their explanations. The proposed relevancy-based XAI method performs the best overall on a variety of metrics. Further investigation on the relative importance of speech/non-speech, phonetic content, and voice onsets/offsets suggest that the XAI results obtained from analyzing limited utterances don't necessarily hold when evaluated on large datasets.

Title: Generating Realistic Forehead-Creases for User Verification via Conditioned Piecewise Polynomial Curves

• Authors: Abhishek Tandon, Geetanjali Sharma, Gaurav Jaswal, Aditya Nigam, Raghavendra Ramachandra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13889
• Pdf URL: https://arxiv.org/pdf/2501.13889
• Copy Paste: [[2501.13889]] Generating Realistic Forehead-Creases for User Verification via Conditioned Piecewise Polynomial Curves(https://arxiv.org/abs/2501.13889)
• Keywords: diffusion
• Abstract: We propose a trait-specific image generation method that models forehead creases geometrically using B-spline and Bézier curves. This approach ensures the realistic generation of both principal creases and non-prominent crease patterns, effectively constructing detailed and authentic forehead-crease images. These geometrically rendered images serve as visual prompts for a diffusion-based Edge-to-Image translation model, which generates corresponding mated samples. The resulting novel synthetic identities are then used to train a forehead-crease verification network. To enhance intra-subject diversity in the generated samples, we employ two strategies: (a) perturbing the control points of B-splines under defined constraints to maintain label consistency, and (b) applying image-level augmentations to the geometric visual prompts, such as dropout and elastic transformations, specifically tailored to crease patterns. By integrating the proposed synthetic dataset with real-world data, our method significantly improves the performance of forehead-crease verification systems under a cross-database verification protocol.

Title: Federated Granger Causality Learning for Interdependent Clients with State Space Representation

• Authors: Ayush Mohanty, Nazal Mohamed, Paritosh Ramanan, Nagi Gebraeel
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.13890
• Pdf URL: https://arxiv.org/pdf/2501.13890
• Copy Paste: [[2501.13890]] Federated Granger Causality Learning for Interdependent Clients with State Space Representation(https://arxiv.org/abs/2501.13890)
• Keywords: security, privacy, robust, federate
• Abstract: Advanced sensors and IoT devices have improved the monitoring and control of complex industrial enterprises. They have also created an interdependent fabric of geographically distributed process operations (clients) across these enterprises. Granger causality is an effective approach to detect and quantify interdependencies by examining how one client's state affects others over time. Understanding these interdependencies captures how localized events, such as faults and disruptions, can propagate throughout the system, possibly causing widespread operational impacts. However, the large volume and complexity of industrial data pose challenges in modeling these interdependencies. This paper develops a federated approach to learning Granger causality. We utilize a linear state space system framework that leverages low-dimensional state estimates to analyze interdependencies. This addresses bandwidth limitations and the computational burden commonly associated with centralized data processing. We propose augmenting the client models with the Granger causality information learned by the server through a Machine Learning (ML) function. We examine the co-dependence between the augmented client and server models and reformulate the framework as a standalone ML algorithm providing conditions for its sublinear and linear convergence rates. We also study the convergence of the framework to a centralized oracle model. Moreover, we include a differential privacy analysis to ensure data security while preserving causal insights. Using synthetic data, we conduct comprehensive experiments to demonstrate the robustness of our approach to perturbations in causality, the scalability to the size of communication, number of clients, and the dimensions of raw data. We also evaluate the performance on two real-world industrial control system datasets by reporting the volume of data saved by decentralization.

Title: Pix2Cap-COCO: Advancing Visual Comprehension via Pixel-Level Captioning

• Authors: Zuyao You, Junke Wang, Lingyu Kong, Bo He, Zuxuan Wu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13893
• Pdf URL: https://arxiv.org/pdf/2501.13893
• Copy Paste: [[2501.13893]] Pix2Cap-COCO: Advancing Visual Comprehension via Pixel-Level Captioning(https://arxiv.org/abs/2501.13893)
• Keywords: robust, segmentation
• Abstract: We present Pix2Cap-COCO, the first panoptic pixel-level caption dataset designed to advance fine-grained visual understanding. To achieve this, we carefully design an automated annotation pipeline that prompts GPT-4V to generate pixel-aligned, instance-specific captions for individual objects within images, enabling models to learn more granular relationships between objects and their contexts. This approach results in 167,254 detailed captions, with an average of 22.94 words per caption. Building on Pix2Cap-COCO, we introduce a novel task, panoptic segmentation-captioning, which challenges models to recognize instances in an image and provide detailed descriptions for each simultaneously. To benchmark this task, we design a robust baseline based on X-Decoder. The experimental results demonstrate that Pix2Cap-COCO is a particularly challenging dataset, as it requires models to excel in both fine-grained visual understanding and detailed language generation. Furthermore, we leverage Pix2Cap-COCO for Supervised Fine-Tuning (SFT) on large multimodal models (LMMs) to enhance their performance. For example, training with Pix2Cap-COCO significantly improves the performance of GPT4RoI, yielding gains in CIDEr +1.4%, ROUGE +0.4%, and SPICE +0.5% on Visual Genome dataset, and strengthens its region understanding ability on the ViP-BENCH, with an overall improvement of +5.1%, including notable increases in recognition accuracy +11.2% and language generation quality +22.2%.

Title: PointOBB-v3: Expanding Performance Boundaries of Single Point-Supervised Oriented Object Detection

• Authors: Peiyuan Zhang, Junwei Luo, Xue Yang, Yi Yu, Qingyun Li, Yue Zhou, Xiaosong Jia, Xudong Lu, Jingdong Chen, Xiang Li, Junchi Yan, Yansheng Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13898
• Pdf URL: https://arxiv.org/pdf/2501.13898
• Copy Paste: [[2501.13898]] PointOBB-v3: Expanding Performance Boundaries of Single Point-Supervised Oriented Object Detection(https://arxiv.org/abs/2501.13898)
• Keywords: fair
• Abstract: With the growing demand for oriented object detection (OOD), recent studies on point-supervised OOD have attracted significant interest. In this paper, we propose PointOBB-v3, a stronger single point-supervised OOD framework. Compared to existing methods, it generates pseudo rotated boxes without additional priors and incorporates support for the end-to-end paradigm. PointOBB-v3 functions by integrating three unique image views: the original view, a resized view, and a rotated/flipped (rot/flp) view. Based on the views, a scale augmentation module and an angle acquisition module are constructed. In the first module, a Scale-Sensitive Consistency (SSC) loss and a Scale-Sensitive Feature Fusion (SSFF) module are introduced to improve the model's ability to estimate object scale. To achieve precise angle predictions, the second module employs symmetry-based self-supervised learning. Additionally, we introduce an end-to-end version that eliminates the pseudo-label generation process by integrating a detector branch and introduces an Instance-Aware Weighting (IAW) strategy to focus on high-quality predictions. We conducted extensive experiments on the DIOR-R, DOTA-v1.0/v1.5/v2.0, FAIR1M, STAR, and RSAR datasets. Across all these datasets, our method achieves an average improvement in accuracy of 3.56% in comparison to previous state-of-the-art methods. The code will be available at this https URL.

Title: Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models

• Authors: Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13904
• Pdf URL: https://arxiv.org/pdf/2501.13904
• Copy Paste: [[2501.13904]] Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models(https://arxiv.org/abs/2501.13904)
• Keywords: privacy, federate, large language model
• Abstract: Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank adaptation scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.

Title: On Learning Representations for Tabular Data Distillation

• Authors: Inwon Kang, Parikshit Ram, Yi Zhou, Horst Samulowitz, Oshani Seneviratne
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13905
• Pdf URL: https://arxiv.org/pdf/2501.13905
• Copy Paste: [[2501.13905]] On Learning Representations for Tabular Data Distillation(https://arxiv.org/abs/2501.13905)
• Keywords: privacy
• Abstract: Dataset distillation generates a small set of information-rich instances from a large dataset, resulting in reduced storage requirements, privacy or copyright risks, and computational costs for downstream modeling, though much of the research has focused on the image data modality. We study tabular data distillation, which brings in novel challenges such as the inherent feature heterogeneity and the common use of non-differentiable learning models (such as decision tree ensembles and nearest-neighbor predictors). To mitigate these challenges, we present $\texttt{TDColER}$, a tabular data distillation framework via column embeddings-based representation learning. To evaluate this framework, we also present a tabular data distillation benchmark, ${\sf \small TDBench}$. Based on an elaborate evaluation on ${\sf \small TDBench}$, resulting in 226,890 distilled datasets and 548,880 models trained on them, we demonstrate that $\texttt{TDColER}$ is able to boost the distilled data quality of off-the-shelf distillation schemes by 0.5-143% across 7 different tabular learning models.

Title: Analysis of Indic Language Capabilities in LLMs

• Authors: Aatman Vaidya, Tarunima Prabhakar, Denny George, Swair Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13912
• Pdf URL: https://arxiv.org/pdf/2501.13912
• Copy Paste: [[2501.13912]] Analysis of Indic Language Capabilities in LLMs(https://arxiv.org/abs/2501.13912)
• Keywords: large language model
• Abstract: This report evaluates the performance of text-in text-out Large Language Models (LLMs) to understand and generate Indic languages. This evaluation is used to identify and prioritize Indic languages suited for inclusion in safety benchmarks. We conduct this study by reviewing existing evaluation studies and datasets; and a set of twenty-eight LLMs that support Indic languages. We analyze the LLMs on the basis of the training data, license for model and data, type of access and model developers. We also compare Indic language performance across evaluation datasets and find that significant performance disparities in performance across Indic languages. Hindi is the most widely represented language in models. While model performance roughly correlates with number of speakers for the top five languages, the assessment after that varies.

Title: Binary Diffusion Probabilistic Model

• Authors: Vitaliy Kinakh, Slava Voloshynovskiy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13915
• Pdf URL: https://arxiv.org/pdf/2501.13915
• Copy Paste: [[2501.13915]] Binary Diffusion Probabilistic Model(https://arxiv.org/abs/2501.13915)
• Keywords: diffusion, generative
• Abstract: We introduce the Binary Diffusion Probabilistic Model (BDPM), a novel generative model optimized for binary data representations. While denoising diffusion probabilistic models (DDPMs) have demonstrated notable success in tasks like image synthesis and restoration, traditional DDPMs rely on continuous data representations and mean squared error (MSE) loss for training, applying Gaussian noise models that may not be optimal for discrete or binary data structures. BDPM addresses this by decomposing images into bitplanes and employing XOR-based noise transformations, with a denoising model trained using binary cross-entropy loss. This approach enables precise noise control and computationally efficient inference, significantly lowering computational costs and improving model convergence. When evaluated on image restoration tasks such as image super-resolution, inpainting, and blind image restoration, BDPM outperforms state-of-the-art methods on the FFHQ, CelebA, and CelebA-HQ datasets. Notably, BDPM requires fewer inference steps than traditional DDPM models to reach optimal results, showcasing enhanced inference efficiency.

Title: PBM-VFL: Vertical Federated Learning with Feature and Sample Privacy

• Authors: Linh Tran, Timothy Castiglia, Stacy Patterson, Ana Milanova
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13916
• Pdf URL: https://arxiv.org/pdf/2501.13916
• Copy Paste: [[2501.13916]] PBM-VFL: Vertical Federated Learning with Feature and Sample Privacy(https://arxiv.org/abs/2501.13916)
• Keywords: secure, privacy, protect, federate
• Abstract: We present Poisson Binomial Mechanism Vertical Federated Learning (PBM-VFL), a communication-efficient Vertical Federated Learning algorithm with Differential Privacy guarantees. PBM-VFL combines Secure Multi-Party Computation with the recently introduced Poisson Binomial Mechanism to protect parties' private datasets during model training. We define the novel concept of feature privacy and analyze end-to-end feature and sample privacy of our algorithm. We compare sample privacy loss in VFL with privacy loss in HFL. We also provide the first theoretical characterization of the relationship between privacy budget, convergence error, and communication cost in differentially-private VFL. Finally, we empirically show that our model performs well with high levels of privacy.

Title: Improving Video Generation with Human Feedback

• Authors: Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, Xintao Wang, Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang, Wanli Ouyang
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13918
• Pdf URL: https://arxiv.org/pdf/2501.13918
• Copy Paste: [[2501.13918]] Improving Video Generation with Human Feedback(https://arxiv.org/abs/2501.13918)
• Keywords: diffusion
• Abstract: Video generation has achieved significant advances through rectified flow techniques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, we begin by constructing a large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models by extending those from diffusion models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and standard supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs. Project page: this https URL.

Title: IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art Text-to-Image Models

• Authors: Jiayi Lei, Renrui Zhang, Xiangfei Hu, Weifeng Lin, Zhen Li, Wenjian Sun, Ruoyi Du, Le Zhuo, Zhongyu Li, Xinyue Li, Shitian Zhao, Ziyu Guo, Yiting Lu, Peng Gao, Hongsheng Li
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13920
• Pdf URL: https://arxiv.org/pdf/2501.13920
• Copy Paste: [[2501.13920]] IMAGINE-E: Image Generation Intelligence Evaluation of State-of-the-art Text-to-Image Models(https://arxiv.org/abs/2501.13920)
• Keywords: diffusion, segmentation
• Abstract: With the rapid development of diffusion models, text-to-image(T2I) models have made significant progress, showcasing impressive abilities in prompt following and image generation. Recently launched models such as FLUX.1 and Ideogram2.0, along with others like Dall-E3 and Stable Diffusion 3, have demonstrated exceptional performance across various complex tasks, raising questions about whether T2I models are moving towards general-purpose applicability. Beyond traditional image generation, these models exhibit capabilities across a range of fields, including controllable generation, image editing, video, audio, 3D, and motion generation, as well as computer vision tasks like semantic segmentation and depth estimation. However, current evaluation frameworks are insufficient to comprehensively assess these models' performance across expanding domains. To thoroughly evaluate these models, we developed the IMAGINE-E and tested six prominent models: FLUX.1, Ideogram2.0, Midjourney, Dall-E3, Stable Diffusion 3, and Jimeng. Our evaluation is divided into five key domains: structured output generation, realism, and physical consistency, specific domain generation, challenging scenario generation, and multi-style creation tasks. This comprehensive assessment highlights each model's strengths and limitations, particularly the outstanding performance of FLUX.1 and Ideogram2.0 in structured and specific domain tasks, underscoring the expanding applications and potential of T2I models as foundational AI tools. This study provides valuable insights into the current state and future trajectory of T2I models as they evolve towards general-purpose usability. Evaluation scripts will be released at this https URL.

Title: Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization

• Authors: Hao Dong, Eleni Chatzi, Olga Fink
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13924
• Pdf URL: https://arxiv.org/pdf/2501.13924
• Copy Paste: [[2501.13924]] Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization(https://arxiv.org/abs/2501.13924)
• Keywords: robust
• Abstract: Test-time adaptation (TTA) has demonstrated significant potential in addressing distribution shifts between training and testing data. Open-set test-time adaptation (OSTTA) aims to adapt a source pre-trained model online to an unlabeled target domain that contains unknown classes. This task becomes more challenging when multiple modalities are involved. Existing methods have primarily focused on unimodal OSTTA, often filtering out low-confidence samples without addressing the complexities of multimodal data. In this work, we present Adaptive Entropy-aware Optimization (AEO), a novel framework specifically designed to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA) for the first time. Our analysis shows that the entropy difference between known and unknown samples in the target domain strongly correlates with MM-OSTTA performance. To leverage this, we propose two key components: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP). These components enhance the ability of model to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples. To thoroughly evaluate our proposed methods in the MM-OSTTA setting, we establish a new benchmark derived from existing datasets. This benchmark includes two downstream tasks and incorporates five modalities. Extensive experiments across various domain shift situations demonstrate the efficacy and versatility of the AEO framework. Additionally, we highlight the strong performance of AEO in long-term and continual MM-OSTTA settings, both of which are challenging and highly relevant to real-world applications. Our source code is available at this https URL.

Title: GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing

• Authors: Akashah Shabbir, Mohammed Zumri, Mohammed Bennamoun, Fahad S. Khan, Salman Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13925
• Pdf URL: https://arxiv.org/pdf/2501.13925
• Copy Paste: [[2501.13925]] GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing(https://arxiv.org/abs/2501.13925)
• Keywords: segmentation
• Abstract: Recent advances in large multimodal models (LMMs) have recognized fine-grained grounding as an imperative factor of visual understanding and dialogue. However, the benefits of such representation in LMMs are limited to the natural image domain, and these models perform poorly for remote sensing (RS). The distinct overhead viewpoint, scale variation, and presence of small objects in high-resolution RS imagery present a unique challenge in region-level comprehension. Moreover, the development of the grounding conversation capability of LMMs within RS is hindered by the lack of granular, RS domain-specific grounded data. Addressing these limitations, we propose GeoPixel - the first end-to-end high resolution RS-LMM that supports pixel-level grounding. This capability allows fine-grained visual perception by generating interleaved masks in conversation. GeoPixel supports up to 4K HD resolution in any aspect ratio, ideal for high-precision RS image analysis. To support the grounded conversation generation (GCG) in RS imagery, we curate a visually grounded dataset GeoPixelD through a semi-automated pipeline that utilizes set-of-marks prompting and spatial priors tailored for RS data to methodically control the data generation process. GeoPixel demonstrates superior performance in pixel-level comprehension, surpassing existing LMMs in both single-target and multi-target segmentation tasks. Our methodological ablation studies validate the effectiveness of each component in the overall architecture. Our code and data will be publicly released.

Title: Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step

• Authors: Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, Pheng-Ann Heng
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.13926
• Pdf URL: https://arxiv.org/pdf/2501.13926
• Copy Paste: [[2501.13926]] Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step(https://arxiv.org/abs/2501.13926)
• Keywords: diffusion
• Abstract: Chain-of-Thought (CoT) reasoning has been extensively explored in large models to tackle complex understanding tasks. However, it still remains an open question whether such strategies can be applied to verifying and reinforcing image generation scenarios. In this paper, we provide the first comprehensive investigation of the potential of CoT reasoning to enhance autoregressive image generation. We focus on three techniques: scaling test-time computation for verification, aligning model preferences with Direct Preference Optimization (DPO), and integrating these techniques for complementary effects. Our results demonstrate that these approaches can be effectively adapted and combined to significantly improve image generation performance. Furthermore, given the pivotal role of reward models in our findings, we propose the Potential Assessment Reward Model (PARM) and PARM++, specialized for autoregressive image generation. PARM adaptively assesses each generation step through a potential assessment approach, merging the strengths of existing reward models, and PARM++ further introduces a reflection mechanism to self-correct the generated unsatisfactory image. Using our investigated reasoning strategies, we enhance a baseline model, Show-o, to achieve superior results, with a significant +24% improvement on the GenEval benchmark, surpassing Stable Diffusion 3 by +15%. We hope our study provides unique insights and paves a new path for integrating CoT reasoning with autoregressive image generation. Code and models are released at this https URL

Title: CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation

• Authors: Guofeng Cui, Pichao Wang, Yang Liu, Zemian Ke, Zhu Liu, Vimal Bhat
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13927
• Pdf URL: https://arxiv.org/pdf/2501.13927
• Copy Paste: [[2501.13927]] CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation(https://arxiv.org/abs/2501.13927)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation (MT) remains challenging due to pretraining on English-centric data and the complexity of reinforcement learning from human feedback (RLHF). Direct Preference Optimization (DPO) has emerged as a simpler and more efficient alternative, but its performance depends heavily on the quality of preference data. To address this, we propose Confidence-Reward driven Preference Optimization (CRPO), a novel method that combines reward scores with model confidence to improve data selection for fine-tuning. CRPO selects challenging sentence pairs where the model is uncertain or underperforms, leading to more effective learning. While primarily designed for LLMs, CRPO also generalizes to encoder-decoder models like NLLB, demonstrating its versatility. Empirical results show that CRPO outperforms existing methods such as RS-DPO, RSO and MBR score in both translation accuracy and data efficiency.

Title: Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass

• Authors: Jianing Yang, Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, Matt Feiszli
• Subjects: cs.CV, cs.AI, cs.GR, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.13928
• Pdf URL: https://arxiv.org/pdf/2501.13928
• Copy Paste: [[2501.13928]] Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass(https://arxiv.org/abs/2501.13928)
• Keywords: robust, transformer
• Abstract: Multi-view 3D reconstruction remains a core challenge in computer vision, particularly in applications requiring accurate and scalable representations across diverse perspectives. Current leading methods such as DUSt3R employ a fundamentally pairwise approach, processing images in pairs and necessitating costly global alignment procedures to reconstruct from multiple views. In this work, we propose Fast 3D Reconstruction (Fast3R), a novel multi-view generalization to DUSt3R that achieves efficient and scalable 3D reconstruction by processing many views in parallel. Fast3R's Transformer-based architecture forwards N images in a single forward pass, bypassing the need for iterative alignment. Through extensive experiments on camera pose estimation and 3D reconstruction, Fast3R demonstrates state-of-the-art performance, with significant improvements in inference speed and reduced error accumulation. These results establish Fast3R as a robust alternative for multi-view applications, offering enhanced scalability without compromising reconstruction accuracy.