2024-09-27

Title: An Art-centric perspective on AI-based content moderation of nudity

• Authors: Piera Riccio, Georgina Curto, Thomas Hofmann, Nuria Oliver
• Subjects: cs.CV, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.17156
• Pdf URL: https://arxiv.org/pdf/2409.17156
• Copy Paste: [[2409.17156]] An Art-centric perspective on AI-based content moderation of nudity(https://arxiv.org/abs/2409.17156)
• Keywords: generative
• Abstract: At a time when the influence of generative Artificial Intelligence on visual arts is a highly debated topic, we raise the attention towards a more subtle phenomenon: the algorithmic censorship of artistic nudity online. We analyze the performance of three "Not-Safe-For-Work'' image classifiers on artistic nudity, and empirically uncover the existence of a gender and a stylistic bias, as well as evident technical limitations, especially when only considering visual information. Hence, we propose a multi-modal zero-shot classification approach that improves artistic nudity classification. From our research, we draw several implications that we hope will inform future research on this topic.

Title: REAL: Response Embedding-based Alignment for LLMs

• Authors: Honggen Zhang, Igor Molybog, June Zhang, Xufeng Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17169
• Pdf URL: https://arxiv.org/pdf/2409.17169
• Copy Paste: [[2409.17169]] REAL: Response Embedding-based Alignment for LLMs(https://arxiv.org/abs/2409.17169)
• Keywords: large language model
• Abstract: Aligning large language models (LLMs) to human preferences is a crucial step in building helpful and safe AI tools, which usually involve training on supervised datasets. Popular algorithms such as Direct Preference Optimization rely on pairs of AI-generated responses ranked according to human feedback. The labeling process is the most labor-intensive and costly part of the alignment pipeline, and improving its efficiency would have a meaningful impact on AI development. We propose a strategy for sampling a high-quality training dataset that focuses on acquiring the most informative response pairs for labeling out of a set of AI-generated responses. Experimental results on synthetic HH-RLHF benchmarks indicate that choosing dissimilar response pairs enhances the direct alignment of LLMs while reducing inherited labeling errors. We also applied our method to the real-world dataset SHP2, selecting optimal pairs from multiple responses. The model aligned on dissimilar response pairs obtained the best win rate on the dialogue task. Our findings suggest that focusing on less similar pairs can improve the efficiency of LLM alignment, saving up to 65% of annotators' work.

Title: What Would You Ask When You First Saw $a^2+b^2=c^2$? Evaluating LLM on Curiosity-Driven Questioning

• Authors: Shashidhar Reddy Javaji, Zining Zhu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17172
• Pdf URL: https://arxiv.org/pdf/2409.17172
• Copy Paste: [[2409.17172]] What Would You Ask When You First Saw $a^2+b^2=c^2$? Evaluating LLM on Curiosity-Driven Questioning(https://arxiv.org/abs/2409.17172)
• Keywords: large language model
• Abstract: Large language models (LLMs) can store a massive amount of knowledge, yet their potential to acquire new knowledge remains unknown. We propose a novel evaluation framework that evaluates this capability. This framework prompts LLMs to generate questions about a statement introducing scientific knowledge, simulating a curious person when facing the statement for the first time. We score the qualities of the generated questions, thereby evaluating the knowledge acquisition potential of the LLM. We apply controlled ablation studies to validate our scoring procedures. Additionally, we created a synthetic dataset consisting of 1101 statements in physics, chemistry, and maths with distinct levels of difficulties, 300 general knowledge statements, and 567 incorrect statements. Human evaluations were conducted to validate our model assessments, achieving an approximate weighted Cohen's kappa of 0.7 on all three metrics considered. We find that while large models like GPT-4 and Mistral 8x7b are adept at generating coherent and relevant questions, the smaller Phi-2 model is equally or more effective. This indicates that size does not solely determine a model's knowledge acquisition potential. The proposed framework quantifies a critical model capability that was commonly overlooked and opens up research opportunities for developing more knowledgeable AI systems

Title: A Multiple-Fill-in-the-Blank Exam Approach for Enhancing Zero-Resource Hallucination Detection in Large Language Models

• Authors: Satoshi Munakata, Taku Fukui, Takao Mohri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17173
• Pdf URL: https://arxiv.org/pdf/2409.17173
• Copy Paste: [[2409.17173]] A Multiple-Fill-in-the-Blank Exam Approach for Enhancing Zero-Resource Hallucination Detection in Large Language Models(https://arxiv.org/abs/2409.17173)
• Keywords: large language model
• Abstract: Large language models (LLMs) often fabricate a hallucinatory text. Several methods have been developed to detect such text by semantically comparing it with the multiple versions probabilistically regenerated. However, a significant issue is that if the storyline of each regenerated text changes, the generated texts become incomparable, which worsen detection accuracy. In this paper, we propose a hallucination detection method that incorporates a multiple-fill-in-the-blank exam approach to address this storyline-changing issue. First, our method creates a multiple-fill-in-the-blank exam by masking multiple objects from the original text. Second, prompts an LLM to repeatedly answer this exam. This approach ensures that the storylines of the exam answers align with the original ones. Finally, quantifies the degree of hallucination for each original sentence by scoring the exam answers, considering the potential for \emph{hallucination snowballing} within the original text itself. Experimental results show that our method alone not only outperforms existing methods, but also achieves clearer state-of-the-art performance in the ensembles with existing methods.

Title: CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Casual Significance and Consistency

• Authors: Kangsheng Wang, Xiao Zhang, Zizheng Guo, Tianyu Hu, Huimin Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17174
• Pdf URL: https://arxiv.org/pdf/2409.17174
• Copy Paste: [[2409.17174]] CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Casual Significance and Consistency(https://arxiv.org/abs/2409.17174)
• Keywords: robust, large language model
• Abstract: Chain-based reasoning methods like chain of thought (CoT) play a rising role in solving reasoning tasks for large language models (LLMs). However, the causal illusions between \textit{a step of reasoning} and \textit{corresponding state transitions} are becoming a significant obstacle to advancing LLMs' reasoning capabilities, especially in long-range reasoning tasks. This paper proposes a non-chain-based reasoning framework for simultaneous consideration of causal significance and consistency, i.e., the Causal Significance and Consistency Enhancer (CSCE). We customize LLM's loss function utilizing treatment effect assessments to enhance its reasoning ability from two aspects: causal significance and consistency. This ensures that the model captures essential causal relationships and maintains robust and consistent performance across various scenarios. Additionally, we transform the reasoning process from the cascading multiple one-step reasoning commonly used in Chain-Based methods, like CoT, to a causal-enhanced method that outputs the entire reasoning process in one go, further improving the model's reasoning efficiency. Extensive experiments show that our method improves both the reasoning success rate and speed. These improvements further demonstrate that non-chain-based methods can also aid LLMs in completing reasoning tasks.

Title: XDC Gasless Subnet: Gasless Subnet Staking dApp for XDC Network

• Authors: Mohuya Chakraborty, Atul Khekade
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17176
• Pdf URL: https://arxiv.org/pdf/2409.17176
• Copy Paste: [[2409.17176]] XDC Gasless Subnet: Gasless Subnet Staking dApp for XDC Network(https://arxiv.org/abs/2409.17176)
• Keywords: security
• Abstract: With a delegated proof-of-stake (XDPoS) consensus mechanism, the XDC Network is an enterprise-focused blockchain platform that combines the strength of public and private blockchains to provide quick transaction times, low energy consumption, and economical gas fees. XDC is designed for interoperability and supports decentralized apps (dApps) and integrates smoothly with financial systems. It is perfect for trade financing and tokenisation of physical assets because of its emphasis on security and scalability. However, there are a few critical issues that hamper wider acceptance and usability for certain high-frequency applications. This whitepaper introduces a novel and enthralling dApp for establishing a gasless subnet in which mainnet XDC can be staked to spin off a subnet that functions similarly to a non-crypto network, accepting currency fees on the XDC network. This would allow users to stake their tokens without incurring gas fees making the staking process more efficient, cost-effective, and simultaneously enhancing scalability. Performance evaluation of the dApp shows promising results in terms of throughput, latency, scalability, security, and cost efficiency. The use cases and applications of this approach along with challenges and ensuing solutions are included.

Title: Fully automatic extraction of morphological traits from the Web: utopia or reality?

• Authors: Diego Marcos, Robert van de Vlasakker, Ioannis N. Athanasiadis, Pierre Bonnet, Hervé Goeau, Alexis Joly, W. Daniel Kissling, César Leblanc, André S.J. van Proosdij, Konstantinos P. Panousis
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17179
• Pdf URL: https://arxiv.org/pdf/2409.17179
• Copy Paste: [[2409.17179]] Fully automatic extraction of morphological traits from the Web: utopia or reality?(https://arxiv.org/abs/2409.17179)
• Keywords: extraction, large language model
• Abstract: Plant morphological traits, their observable characteristics, are fundamental to understand the role played by each species within their ecosystem. However, compiling trait information for even a moderate number of species is a demanding task that may take experts years to accomplish. At the same time, massive amounts of information about species descriptions is available online in the form of text, although the lack of structure makes this source of data impossible to use at scale. To overcome this, we propose to leverage recent advances in large language models (LLMs) and devise a mechanism for gathering and processing information on plant traits in the form of unstructured textual descriptions, without manual curation. We evaluate our approach by automatically replicating three manually created species-trait matrices. Our method managed to find values for over half of all species-trait pairs, with an F1-score of over 75%. Our results suggest that large-scale creation of structured trait databases from unstructured online text is currently feasible thanks to the information extraction capabilities of LLMs, being limited by the availability of textual descriptions covering all the traits of interest.

Title: A Mobile Payment Scheme Using Biometric Identification with Mutual Authentication

• Authors: Jack Sturgess, Ivan Martinovic
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17181
• Pdf URL: https://arxiv.org/pdf/2409.17181
• Copy Paste: [[2409.17181]] A Mobile Payment Scheme Using Biometric Identification with Mutual Authentication(https://arxiv.org/abs/2409.17181)
• Keywords: protect, attack, biometric
• Abstract: Cashless payment systems offer many benefits over cash, but also have some drawbacks. Fake terminals, skimming, wireless connectivity, and relay attacks are persistent problems. Attempts to overcome one problem often lead to another - for example, some systems use QR codes to avoid skimming and connexion issues, but QR codes can be stolen at distance and relayed. In this paper, we propose a novel mobile payment scheme based on biometric identification that provides mutual authentication to protect the user from rogue terminals. Our scheme imposes only minimal requirements on terminal hardware, does not depend on wireless connectivity between the user and the verifier during the authentication phase, and does not require the user to trust the terminal until it has authenticated itself to the user. We show that our scheme is resistant against phishing, replay, relay, and presentation attacks.

Title: Enhancing Guardrails for Safe and Secure Healthcare AI

• Authors: Ananya Gangavarapu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17190
• Pdf URL: https://arxiv.org/pdf/2409.17190
• Copy Paste: [[2409.17190]] Enhancing Guardrails for Safe and Secure Healthcare AI(https://arxiv.org/abs/2409.17190)
• Keywords: secure, security, robust, generative
• Abstract: Generative AI holds immense promise in addressing global healthcare access challenges, with numerous innovative applications now ready for use across various healthcare domains. However, a significant barrier to the widespread adoption of these domain-specific AI solutions is the lack of robust safety mechanisms to effectively manage issues such as hallucination, misinformation, and ensuring truthfulness. Left unchecked, these risks can compromise patient safety and erode trust in healthcare AI systems. While general-purpose frameworks like Llama Guard are useful for filtering toxicity and harmful content, they do not fully address the stringent requirements for truthfulness and safety in healthcare contexts. This paper examines the unique safety and security challenges inherent to healthcare AI, particularly the risk of hallucinations, the spread of misinformation, and the need for factual accuracy in clinical settings. I propose enhancements to existing guardrails frameworks, such as Nvidia NeMo Guardrails, to better suit healthcare-specific needs. By strengthening these safeguards, I aim to ensure the secure, reliable, and accurate use of AI in healthcare, mitigating misinformation risks and improving patient safety.

Title: An Effective, Robust and Fairness-aware Hate Speech Detection Framework

• Authors: Guanyi Mou, Kyumin Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17191
• Pdf URL: https://arxiv.org/pdf/2409.17191
• Copy Paste: [[2409.17191]] An Effective, Robust and Fairness-aware Hate Speech Detection Framework(https://arxiv.org/abs/2409.17191)
• Keywords: attack, robust, fair
• Abstract: With the widespread online social networks, hate speeches are spreading faster and causing more damage than ever before. Existing hate speech detection methods have limitations in several aspects, such as handling data insufficiency, estimating model uncertainty, improving robustness against malicious attacks, and handling unintended bias (i.e., fairness). There is an urgent need for accurate, robust, and fair hate speech classification in online social networks. To bridge the gap, we design a data-augmented, fairness addressed, and uncertainty estimated novel framework. As parts of the framework, we propose Bidirectional Quaternion-Quasi-LSTM layers to balance effectiveness and efficiency. To build a generalized model, we combine five datasets collected from three platforms. Experiment results show that our model outperforms eight state-of-the-art methods under both no attack scenario and various attack scenarios, indicating the effectiveness and robustness of our model. We share our code along with combined dataset for better future research

Title: A random measure approach to reinforcement learning in continuous time

• Authors: Christian Bender, Nguyen Tran Thuan
• Subjects: cs.LG, math.PR, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.17200
• Pdf URL: https://arxiv.org/pdf/2409.17200
• Copy Paste: [[2409.17200]] A random measure approach to reinforcement learning in continuous time(https://arxiv.org/abs/2409.17200)
• Keywords: diffusion
• Abstract: We present a random measure approach for modeling exploration, i.e., the execution of measure-valued controls, in continuous-time reinforcement learning (RL) with controlled diffusion and jumps. First, we consider the case when sampling the randomized control in continuous time takes place on a discrete-time grid and reformulate the resulting stochastic differential equation (SDE) as an equation driven by suitable random measures. The construction of these random measures makes use of the Brownian motion and the Poisson random measure (which are the sources of noise in the original model dynamics) as well as the additional random variables, which are sampled on the grid for the control execution. Then, we prove a limit theorem for these random measures as the mesh-size of the sampling grid goes to zero, which leads to the grid-sampling limit SDE that is jointly driven by white noise random measures and a Poisson random measure. We also argue that the grid-sampling limit SDE can substitute the exploratory SDE and the sample SDE of the recent continuous-time RL literature, i.e., it can be applied for the theoretical analysis of exploratory control problems and for the derivation of learning algorithms.

Title: Immersion and Invariance-based Coding for Privacy-Preserving Federated Learning

• Authors: Haleh Hayati, Carlos Murguia, Nathan van de Wouw
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17201
• Pdf URL: https://arxiv.org/pdf/2409.17201
• Copy Paste: [[2409.17201]] Immersion and Invariance-based Coding for Privacy-Preserving Federated Learning(https://arxiv.org/abs/2409.17201)
• Keywords: privacy, protect, federate
• Abstract: Federated learning (FL) has emerged as a method to preserve privacy in collaborative distributed learning. In FL, clients train AI models directly on their devices rather than sharing data with a centralized server, which can pose privacy risks. However, it has been shown that despite FL's partial protection of local data privacy, information about clients' data can still be inferred from shared model updates during training. In recent years, several privacy-preserving approaches have been developed to mitigate this privacy leakage in FL, though they often provide privacy at the cost of model performance or system efficiency. Balancing these trade-offs presents a significant challenge in implementing FL schemes. In this manuscript, we introduce a privacy-preserving FL framework that combines differential privacy and system immersion tools from control theory. The core idea is to treat the optimization algorithms used in standard FL schemes (e.g., gradient-based algorithms) as a dynamical system that we seek to immerse into a higher-dimensional system (referred to as the target optimization algorithm). The target algorithm's dynamics are designed such that, first, the model parameters of the original algorithm are immersed in its parameters; second, it operates on distorted parameters; and third, it converges to an encoded version of the true model parameters from the original algorithm. These encoded parameters can then be decoded at the server to retrieve the original model parameters. We demonstrate that the proposed privacy-preserving scheme can be tailored to offer any desired level of differential privacy for both local and global model parameters, while maintaining the same accuracy and convergence rate as standard FL algorithms.

Title: 2024 BRAVO Challenge Track 1 1st Place Report: Evaluating Robustness of Vision Foundation Models for Semantic Segmentation

• Authors: Tommie Kerssies, Daan de Geus, Gijs Dubbelman
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.17208
• Pdf URL: https://arxiv.org/pdf/2409.17208
• Copy Paste: [[2409.17208]] 2024 BRAVO Challenge Track 1 1st Place Report: Evaluating Robustness of Vision Foundation Models for Semantic Segmentation(https://arxiv.org/abs/2409.17208)
• Keywords: robust, segmentation
• Abstract: In this report, we present our solution for Track 1 of the 2024 BRAVO Challenge, where a model is trained on Cityscapes and its robustness is evaluated on several out-of-distribution datasets. Our solution leverages the powerful representations learned by vision foundation models, by attaching a simple segmentation decoder to DINOv2 and fine-tuning the entire model. This approach outperforms more complex existing approaches, and achieves 1st place in the challenge. Our code is publicly available at this https URL.

Title: Mnemosyne: Parallelization Strategies for Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations

• Authors: Amey Agrawal, Junda Chen, Íñigo Goiri, Ramachandran Ramjee, Chaojie Zhang, Alexey Tumanov, Esha Choukse
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.17264
• Pdf URL: https://arxiv.org/pdf/2409.17264
• Copy Paste: [[2409.17264]] Mnemosyne: Parallelization Strategies for Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations(https://arxiv.org/abs/2409.17264)
• Keywords: large language model
• Abstract: As large language models (LLMs) evolve to handle increasingly longer contexts, serving inference requests for context lengths in the range of millions of tokens presents unique challenges. While existing techniques are effective for training, they fail to address the unique challenges of inference, such as varying prefill and decode phases and their associated latency constraints - like Time to First Token (TTFT) and Time Between Tokens (TBT). Furthermore, there are no long context inference solutions that allow batching requests to increase the hardware utilization today. In this paper, we propose three key innovations for efficient interactive long context LLM inference, without resorting to any approximation: adaptive chunking to reduce prefill overheads in mixed batching, Sequence Pipeline Parallelism (SPP) to lower TTFT, and KV Cache Parallelism (KVP) to minimize TBT. These contributions are combined into a 3D parallelism strategy, enabling Mnemosyne to scale interactive inference to context lengths at least up to 10 million tokens with high throughput enabled with batching. To our knowledge, Mnemosyne is the first to be able to achieve support for 10 million long context inference efficiently, while satisfying production-grade SLOs on TBT (30ms) on contexts up to and including 10 million.

Title: Model aggregation: minimizing empirical variance outperforms minimizing empirical error

• Authors: Théo Bourdais, Houman Owhadi
• Subjects: cs.LG, cs.AI, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.17267
• Pdf URL: https://arxiv.org/pdf/2409.17267
• Copy Paste: [[2409.17267]] Model aggregation: minimizing empirical variance outperforms minimizing empirical error(https://arxiv.org/abs/2409.17267)
• Keywords: robust
• Abstract: Whether deterministic or stochastic, models can be viewed as functions designed to approximate a specific quantity of interest. We propose a data-driven framework that aggregates predictions from diverse models into a single, more accurate output. This aggregation approach exploits each model's strengths to enhance overall accuracy. It is non-intrusive - treating models as black-box functions - model-agnostic, requires minimal assumptions, and can combine outputs from a wide range of models, including those from machine learning and numerical solvers. We argue that the aggregation process should be point-wise linear and propose two methods to find an optimal aggregate: Minimal Error Aggregation (MEA), which minimizes the aggregate's prediction error, and Minimal Variance Aggregation (MVA), which minimizes its variance. While MEA is inherently more accurate when correlations between models and the target quantity are perfectly known, Minimal Empirical Variance Aggregation (MEVA), an empirical version of MVA - consistently outperforms Minimal Empirical Error Aggregation (MEEA), the empirical counterpart of MEA, when these correlations must be estimated from data. The key difference is that MEVA constructs an aggregate by estimating model errors, while MEEA treats the models as features for direct interpolation of the quantity of interest. This makes MEEA more susceptible to overfitting and poor generalization, where the aggregate may underperform individual models during testing. We demonstrate the versatility and effectiveness of our framework in various applications, such as data science and partial differential equations, showing how it successfully integrates traditional solvers with machine learning models to improve both robustness and accuracy.

Title: On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains

• Authors: Xun Xian, Ganghua Wang, Xuan Bi, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, Jie Ding
• Subjects: cs.CR, cs.AI, cs.CL, cs.DB, cs.ET, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17275
• Pdf URL: https://arxiv.org/pdf/2409.17275
• Copy Paste: [[2409.17275]] On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains(https://arxiv.org/abs/2409.17275)
• Keywords: defense, attack, robust, large language model
• Abstract: Retrieval-Augmented Generation (RAG) has been empirically shown to enhance the performance of large language models (LLMs) in knowledge-intensive domains such as healthcare, finance, and legal contexts. Given a query, RAG retrieves relevant documents from a corpus and integrates them into the LLMs' generation process. In this study, we investigate the adversarial robustness of RAG, focusing specifically on examining the retrieval system. First, across 225 different setup combinations of corpus, retriever, query, and targeted information, we show that retrieval systems are vulnerable to universal poisoning attacks in medical Q\&A. In such attacks, adversaries generate poisoned documents containing a broad spectrum of targeted information, such as personally identifiable information. When these poisoned documents are inserted into a corpus, they can be accurately retrieved by any users, as long as attacker-specified queries are used. To understand this vulnerability, we discovered that the deviation from the query's embedding to that of the poisoned document tends to follow a pattern in which the high similarity between the poisoned document and the query is retained, thereby enabling precise retrieval. Based on these findings, we develop a new detection-based defense to ensure the safe use of RAG. Through extensive experiments spanning various Q\&A domains, we observed that our proposed method consistently achieves excellent detection rates in nearly all cases.

Title: SHEATH: Defending Horizontal Collaboration for Distributed CNNs against Adversarial Noise

• Authors: Muneeba Asif, Mohammad Kumail Kazmi, Mohammad Ashiqur Rahman, Syed Rafay Hasan, Soamar Homsi
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.17279
• Pdf URL: https://arxiv.org/pdf/2409.17279
• Copy Paste: [[2409.17279]] SHEATH: Defending Horizontal Collaboration for Distributed CNNs against Adversarial Noise(https://arxiv.org/abs/2409.17279)
• Keywords: secure, security, privacy, attack
• Abstract: As edge computing and the Internet of Things (IoT) expand, horizontal collaboration (HC) emerges as a distributed data processing solution for resource-constrained devices. In particular, a convolutional neural network (CNN) model can be deployed on multiple IoT devices, allowing distributed inference execution for image recognition while ensuring model and data privacy. Yet, this distributed architecture remains vulnerable to adversaries who want to make subtle alterations that impact the model, even if they lack access to the entire model. Such vulnerabilities can have severe implications for various sectors, including healthcare, military, and autonomous systems. However, security solutions for these vulnerabilities have not been explored. This paper presents a novel framework for Secure Horizontal Edge with Adversarial Threat Handling (SHEATH) to detect adversarial noise and eliminate its effect on CNN inference by recovering the original feature maps. Specifically, SHEATH aims to address vulnerabilities without requiring complete knowledge of the CNN model in HC edge architectures based on sequential partitioning. It ensures data and model integrity, offering security against adversarial attacks in diverse HC environments. Our evaluations demonstrate SHEATH's adaptability and effectiveness across diverse CNN configurations.

Title: Disco4D: Disentangled 4D Human Generation and Animation from a Single Image

• Authors: Hui En Pang, Shuai Liu, Zhongang Cai, Lei Yang, Tianwei Zhang, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17280
• Pdf URL: https://arxiv.org/pdf/2409.17280
• Copy Paste: [[2409.17280]] Disco4D: Disentangled 4D Human Generation and Animation from a Single Image(https://arxiv.org/abs/2409.17280)
• Keywords: extraction, diffusion
• Abstract: We present \textbf{Disco4D}, a novel Gaussian Splatting framework for 4D human generation and animation from a single image. Different from existing methods, Disco4D distinctively disentangles clothings (with Gaussian models) from the human body (with SMPL-X model), significantly enhancing the generation details and flexibility. It has the following technical innovations. \textbf{1)} Disco4D learns to efficiently fit the clothing Gaussians over the SMPL-X Gaussians. \textbf{2)} It adopts diffusion models to enhance the 3D generation process, \textit{e.g.}, modeling occluded parts not visible in the input image. \textbf{3)} It learns an identity encoding for each clothing Gaussian to facilitate the separation and extraction of clothing assets. Furthermore, Disco4D naturally supports 4D human animation with vivid dynamics. Extensive experiments demonstrate the superiority of Disco4D on 4D human generation and animation tasks. Our visualizations can be found in \url{this https URL}.

Title: Investigating Privacy Attacks in the Gray-Box Setting to Enhance Collaborative Learning Schemes

• Authors: Federico Mazzone, Ahmad Al Badawi, Yuriy Polyakov, Maarten Everts, Florian Hahn, Andreas Peter
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17283
• Pdf URL: https://arxiv.org/pdf/2409.17283
• Copy Paste: [[2409.17283]] Investigating Privacy Attacks in the Gray-Box Setting to Enhance Collaborative Learning Schemes(https://arxiv.org/abs/2409.17283)
• Keywords: privacy, protect, attack
• Abstract: The notion that collaborative machine learning can ensure privacy by just withholding the raw data is widely acknowledged to be flawed. Over the past seven years, the literature has revealed several privacy attacks that enable adversaries to extract information about a model's training dataset by exploiting access to model parameters during or after training. In this work, we study privacy attacks in the gray-box setting, where the attacker has only limited access - in terms of view and actions - to the model. The findings of our investigation provide new insights for the development of privacy-preserving collaborative learning solutions. We deploy SmartCryptNN, a framework that tailors homomorphic encryption to protect the portions of the model posing higher privacy risks. Our solution offers a trade-off between privacy and efficiency, which varies based on the extent and selection of the model components we choose to protect. We explore it on dense neural networks, where through extensive evaluation of diverse datasets and architectures, we uncover instances where a favorable sweet spot in the trade-off can be achieved by safeguarding only a single layer of the network. In one of such instances, our approach trains ~4 times faster compared to fully encrypted solutions, while reducing membership leakage by 17.8 times compared to plaintext solutions.

Title: Blockchain-Enabled Variational Information Bottleneck for Data Extraction Based on Mutual Information in Internet of Vehicles

• Authors: Cui Zhang, Wenjun Zhang, Qiong Wu, Pingyi Fan, Nan Cheng, Wen Chen, Khaled B. Letaief
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17287
• Pdf URL: https://arxiv.org/pdf/2409.17287
• Copy Paste: [[2409.17287]] Blockchain-Enabled Variational Information Bottleneck for Data Extraction Based on Mutual Information in Internet of Vehicles(https://arxiv.org/abs/2409.17287)
• Keywords: secure, security, privacy, extraction
• Abstract: The Internet of Vehicles (IoV) network can address the issue of limited computing resources and data processing capabilities of individual vehicles, but it also brings the risk of privacy leakage to vehicle users. Applying blockchain technology can establish secure data links within the IoV, solving the problems of insufficient computing resources for each vehicle and the security of data transmission over the network. However, with the development of the IoV, the amount of data interaction between multiple vehicles and between vehicles and base stations, roadside units, etc., is continuously increasing. There is a need to further reduce the interaction volume, and intelligent data compression is key to solving this problem. The VIB technique facilitates the training of encoding and decoding models, substantially diminishing the volume of data that needs to be transmitted. This paper introduces an innovative approach that integrates blockchain with VIB, referred to as BVIB, designed to lighten computational workloads and reinforce the security of the network. We first construct a new network framework by separating the encoding and decoding networks to address the computational burden issue, and then propose a new algorithm to enhance the security of IoV networks. We also discuss the impact of the data extraction rate on system latency to determine the most suitable data extraction rate. An experimental framework combining Python and C++ has been established to substantiate the efficacy of our BVIB approach. Comprehensive simulation studies indicate that the BVIB consistently excels in comparison to alternative foundational methodologies.

Title: Consistent estimation of generative model representations in the data kernel perspective space

• Authors: Aranyak Acharyya, Michael W. Trosset, Carey E. Priebe, Hayden S. Helm
• Subjects: cs.LG, math.ST
• Abstract URL: https://arxiv.org/abs/2409.17308
• Pdf URL: https://arxiv.org/pdf/2409.17308
• Copy Paste: [[2409.17308]] Consistent estimation of generative model representations in the data kernel perspective space(https://arxiv.org/abs/2409.17308)
• Keywords: diffusion, generative, large language model
• Abstract: Generative models, such as large language models and text-to-image diffusion models, produce relevant information when presented a query. Different models may produce different information when presented the same query. As the landscape of generative models evolves, it is important to develop techniques to study and analyze differences in model behaviour. In this paper we present novel theoretical results for embedding-based representations of generative models in the context of a set of queries. We establish sufficient conditions for the consistent estimation of the model embeddings in situations where the query set and the number of models grow.

Title: KIPPS: Knowledge infusion in Privacy Preserving Synthetic Data Generation

• Authors: Anantaa Kotal, Anupam Joshi
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17315
• Pdf URL: https://arxiv.org/pdf/2409.17315
• Copy Paste: [[2409.17315]] KIPPS: Knowledge infusion in Privacy Preserving Synthetic Data Generation(https://arxiv.org/abs/2409.17315)
• Keywords: security, privacy, generative
• Abstract: The integration of privacy measures, including differential privacy techniques, ensures a provable privacy guarantee for the synthetic data. However, challenges arise for Generative Deep Learning models when tasked with generating realistic data, especially in critical domains such as Cybersecurity and Healthcare. Generative Models optimized for continuous data struggle to model discrete and non-Gaussian features that have domain constraints. Challenges increase when the training datasets are limited and not diverse. In such cases, generative models create synthetic data that repeats sensitive features, which is a privacy risk. Moreover, generative models face difficulties comprehending attribute constraints in specialized domains. This leads to the generation of unrealistic data that impacts downstream accuracy. To address these issues, this paper proposes a novel model, KIPPS, that infuses Domain and Regulatory Knowledge from Knowledge Graphs into Generative Deep Learning models for enhanced Privacy Preserving Synthetic data generation. The novel framework augments the training of generative models with supplementary context about attribute values and enforces domain constraints during training. This added guidance enhances the model's capacity to generate realistic and domain-compliant synthetic data. The proposed model is evaluated on real-world datasets, specifically in the domains of Cybersecurity and Healthcare, where domain constraints and rules add to the complexity of the data. Our experiments evaluate the privacy resilience and downstream accuracy of the model against benchmark methods, demonstrating its effectiveness in addressing the balance between privacy preservation and data accuracy in complex domains.

Title: Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological Measurement

• Authors: Haodong Li, Hao Lu, Ying-Cong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17316
• Pdf URL: https://arxiv.org/pdf/2409.17316
• Copy Paste: [[2409.17316]] Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological Measurement(https://arxiv.org/abs/2409.17316)
• Keywords: privacy
• Abstract: Remote photoplethysmography (rPPG) is gaining prominence for its non-invasive approach to monitoring physiological signals using only cameras. Despite its promise, the adaptability of rPPG models to new, unseen domains is hindered due to the environmental sensitivity of physiological signals. To address this, we pioneer the Test-Time Adaptation (TTA) in rPPG, enabling the adaptation of pre-trained models to the target domain during inference, sidestepping the need for annotations or source data due to privacy considerations. Particularly, utilizing only the user's face video stream as the accessible target domain data, the rPPG model is adjusted by tuning on each single instance it encounters. However, 1) TTA algorithms are designed predominantly for classification tasks, ill-suited in regression tasks such as rPPG due to inadequate supervision. 2) Tuning pre-trained models in a single-instance manner introduces variability and instability, posing challenges to effectively filtering domain-relevant from domain-irrelevant features while simultaneously preserving the learned information. To overcome these challenges, we present Bi-TTA, a novel expert knowledge-based Bidirectional Test-Time Adapter framework. Specifically, leveraging two expert-knowledge priors for providing self-supervision, our Bi-TTA primarily comprises two modules: a prospective adaptation (PA) module using sharpness-aware minimization to eliminate domain-irrelevant noise, enhancing the stability and efficacy during the adaptation process, and a retrospective stabilization (RS) module to dynamically reinforce crucial learned model parameters, averting performance degradation caused by overfitting or catastrophic forgetting. To this end, we established a large-scale benchmark for rPPG tasks under TTA protocol. The experimental results demonstrate the significant superiority of our approach over the state-of-the-art.

Title: The poison of dimensionality

• Authors: Lê-Nguyên Hoang
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.17328
• Pdf URL: https://arxiv.org/pdf/2409.17328
• Copy Paste: [[2409.17328]] The poison of dimensionality(https://arxiv.org/abs/2409.17328)
• Keywords: defense, attack, robust
• Abstract: This paper advances the understanding of how the size of a machine learning model affects its vulnerability to poisoning, despite state-of-the-art defenses. Given isotropic random honest feature vectors and the geometric median (or clipped mean) as the robust gradient aggregator rule, we essentially prove that, perhaps surprisingly, linear and logistic regressions with $D \geq 169 H^2/P^2$ parameters are subject to arbitrary model manipulation by poisoners, where $H$ and $P$ are the numbers of honestly labeled and poisoned data points used for training. Our experiments go on exposing a fundamental tradeoff between augmenting model expressivity and increasing the poisoners' attack surface, on both synthetic data, and on MNIST & FashionMNIST data for linear classifiers with random features. We also discuss potential implications for source-based learning and neural nets.

Title: VL4AD: Vision-Language Models Improve Pixel-wise Anomaly Detection

• Authors: Liangyu Zhong, Joachim Sicking, Fabian Hüger, Hanno Gottschalk
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17330
• Pdf URL: https://arxiv.org/pdf/2409.17330
• Copy Paste: [[2409.17330]] VL4AD: Vision-Language Models Improve Pixel-wise Anomaly Detection(https://arxiv.org/abs/2409.17330)
• Keywords: segmentation
• Abstract: Semantic segmentation networks have achieved significant success under the assumption of independent and identically distributed data. However, these networks often struggle to detect anomalies from unknown semantic classes due to the limited set of visual concepts they are typically trained on. To address this issue, anomaly segmentation often involves fine-tuning on outlier samples, necessitating additional efforts for data collection, labeling, and model retraining. Seeking to avoid this cumbersome work, we take a different approach and propose to incorporate Vision-Language (VL) encoders into existing anomaly detectors to leverage the semantically broad VL pre-training for improved outlier awareness. Additionally, we propose a new scoring function that enables data- and training-free outlier supervision via textual prompts. The resulting VL4AD model, which includes max-logit prompt ensembling and a class-merging strategy, achieves competitive performance on widely used benchmark datasets, thereby demonstrating the potential of vision-language models for pixel-wise anomaly detection.

Title: ChatCam: Empowering Camera Control through Conversational AI

• Authors: Xinhang Liu, Yu-Wing Tai, Chi-Keung Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17331
• Pdf URL: https://arxiv.org/pdf/2409.17331
• Copy Paste: [[2409.17331]] ChatCam: Empowering Camera Control through Conversational AI(https://arxiv.org/abs/2409.17331)
• Keywords: large language model
• Abstract: Cinematographers adeptly capture the essence of the world, crafting compelling visual narratives through intricate camera movements. Witnessing the strides made by large language models in perceiving and interacting with the 3D world, this study explores their capability to control cameras with human language guidance. We introduce ChatCam, a system that navigates camera movements through conversations with users, mimicking a professional cinematographer's workflow. To achieve this, we propose CineGPT, a GPT-based autoregressive model for text-conditioned camera trajectory generation. We also develop an Anchor Determinator to ensure precise camera trajectory placement. ChatCam understands user requests and employs our proposed tools to generate trajectories, which can be used to render high-quality video footage on radiance field representations. Our experiments, including comparisons to state-of-the-art approaches and user studies, demonstrate our approach's ability to interpret and execute complex instructions for camera operation, showing promising applications in real-world production settings.

Title: Block Expanded DINORET: Adapting Natural Domain Foundation Models for Retinal Imaging Without Catastrophic Forgetting

• Authors: Jay Zoellin, Colin Merk, Mischa Buob, Amr Saad, Samuel Giesser, Tahm Spitznagel, Ferhat Turgut, Rui Santos, Yukun Zhou, Sigfried Wagner, Pearse A. Keane, Yih Chung Tham, Delia Cabrera DeBuc, Matthias D. Becker, Gabor M. Somfai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17332
• Pdf URL: https://arxiv.org/pdf/2409.17332
• Copy Paste: [[2409.17332]] Block Expanded DINORET: Adapting Natural Domain Foundation Models for Retinal Imaging Without Catastrophic Forgetting(https://arxiv.org/abs/2409.17332)
• Keywords: robust, transformer
• Abstract: Integrating deep learning into medical imaging is poised to greatly advance diagnostic methods but it faces challenges with generalizability. Foundation models, based on self-supervised learning, address these issues and improve data efficiency. Natural domain foundation models show promise for medical imaging, but systematic research evaluating domain adaptation, especially using self-supervised learning and parameter-efficient fine-tuning, remains underexplored. Additionally, little research addresses the issue of catastrophic forgetting during fine-tuning of foundation models. We adapted the DINOv2 vision transformer for retinal imaging classification tasks using self-supervised learning and generated two novel foundation models termed DINORET and BE DINORET. Publicly available color fundus photographs were employed for model development and subsequent fine-tuning for diabetic retinopathy staging and glaucoma detection. We introduced block expansion as a novel domain adaptation strategy and assessed the models for catastrophic forgetting. Models were benchmarked to RETFound, a state-of-the-art foundation model in ophthalmology. DINORET and BE DINORET demonstrated competitive performance on retinal imaging tasks, with the block expanded model achieving the highest scores on most datasets. Block expansion successfully mitigated catastrophic forgetting. Our few-shot learning studies indicated that DINORET and BE DINORET outperform RETFound in terms of data-efficiency. This study highlights the potential of adapting natural domain vision models to retinal imaging using self-supervised learning and block expansion. BE DINORET offers robust performance without sacrificing previously acquired capabilities. Our findings suggest that these methods could enable healthcare institutions to develop tailored vision models for their patient populations, enhancing global healthcare inclusivity.

Title: Non-asymptotic Convergence of Training Transformers for Next-token Prediction

• Authors: Ruiquan Huang, Yingbin Liang, Jing Yang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.17335
• Pdf URL: https://arxiv.org/pdf/2409.17335
• Copy Paste: [[2409.17335]] Non-asymptotic Convergence of Training Transformers for Next-token Prediction(https://arxiv.org/abs/2409.17335)
• Keywords: transformer
• Abstract: Transformers have achieved extraordinary success in modern machine learning due to their excellent ability to handle sequential data, especially in next-token prediction (NTP) tasks. However, the theoretical understanding of their performance in NTP is limited, with existing studies focusing mainly on asymptotic performance. This paper provides a fine-grained non-asymptotic analysis of the training dynamics of a one-layer transformer consisting of a self-attention module followed by a feed-forward layer. We first characterize the essential structural properties of training datasets for NTP using a mathematical framework based on partial orders. Then, we design a two-stage training algorithm, where the pre-processing stage for training the feed-forward layer and the main stage for training the attention layer exhibit fast convergence performance. Specifically, both layers converge sub-linearly to the direction of their corresponding max-margin solutions. We also show that the cross-entropy loss enjoys a linear convergence rate. Furthermore, we show that the trained transformer presents non-trivial prediction ability with dataset shift, which sheds light on the remarkable generalization performance of transformers. Our analysis technique involves the development of novel properties on the attention gradient and further in-depth analysis of how these properties contribute to the convergence of the training process. Our experiments further validate our theoretical findings.

Title: A vision-based framework for human behavior understanding in industrial assembly lines

• Authors: Konstantinos Papoutsakis, Nikolaos Bakalos, Konstantinos Fragkoulis, Athena Zacharia, Georgia Kapetadimitri, Maria Pateraki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17356
• Pdf URL: https://arxiv.org/pdf/2409.17356
• Copy Paste: [[2409.17356]] A vision-based framework for human behavior understanding in industrial assembly lines(https://arxiv.org/abs/2409.17356)
• Keywords: robust
• Abstract: This paper introduces a vision-based framework for capturing and understanding human behavior in industrial assembly lines, focusing on car door manufacturing. The framework leverages advanced computer vision techniques to estimate workers' locations and 3D poses and analyze work postures, actions, and task progress. A key contribution is the introduction of the CarDA dataset, which contains domain-relevant assembly actions captured in a realistic setting to support the analysis of the framework for human pose and action analysis. The dataset comprises time-synchronized multi-camera RGB-D videos, motion capture data recorded in a real car manufacturing environment, and annotations for EAWS-based ergonomic risk scores and assembly activities. Experimental results demonstrate the effectiveness of the proposed approach in classifying worker postures and robust performance in monitoring assembly task progress.

Title: Improving satellite imagery segmentation using multiple Sentinel-2 revisits

• Authors: Kartik Jindgar, Grace W. Lindsay
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17363
• Pdf URL: https://arxiv.org/pdf/2409.17363
• Copy Paste: [[2409.17363]] Improving satellite imagery segmentation using multiple Sentinel-2 revisits(https://arxiv.org/abs/2409.17363)
• Keywords: transformer, segmentation
• Abstract: In recent years, analysis of remote sensing data has benefited immensely from borrowing techniques from the broader field of computer vision, such as the use of shared models pre-trained on large and diverse datasets. However, satellite imagery has unique features that are not accounted for in traditional computer vision, such as the existence of multiple revisits of the same location. Here, we explore the best way to use revisits in the framework of fine-tuning pre-trained remote sensing models. We focus on an applied research question of relevance to climate change mitigation -- power substation segmentation -- that is representative of applied uses of pre-trained models more generally. Through extensive tests of different multi-temporal input schemes across diverse model architectures, we find that fusing representations from multiple revisits in the model latent space is superior to other methods of using revisits, including as a form of data augmentation. We also find that a SWIN Transformer-based architecture performs better than U-nets and ViT-based models. We verify the generality of our results on a separate building density estimation task.

Title: The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach

• Authors: David Bertoin, Eduardo Hugo Sanchez, Mehdi Zouitine, Emmanuel Rachelson
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17370
• Pdf URL: https://arxiv.org/pdf/2409.17370
• Copy Paste: [[2409.17370]] The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach(https://arxiv.org/abs/2409.17370)
• Keywords: transformer
• Abstract: Despite transformers being considered as the new standard in computer vision, convolutional neural networks (CNNs) still outperform them in low-data regimes. Nonetheless, CNNs often make decisions based on narrow, specific regions of input images, especially when training data is limited. This behavior can severely compromise the model's generalization capabilities, making it disproportionately dependent on certain features that might not represent the broader context of images. While the conditions leading to this phenomenon remain elusive, the primary intent of this article is to shed light on this observed behavior of neural networks. Our research endeavors to prioritize comprehensive insight and to outline an initial response to this phenomenon. In line with this, we introduce Saliency Guided Dropout (SGDrop), a pioneering regularization approach tailored to address this specific issue. SGDrop utilizes attribution methods on the feature map to identify and then reduce the influence of the most salient features during training. This process encourages the network to diversify its attention and not focus solely on specific standout areas. Our experiments across several visual classification benchmarks validate SGDrop's role in enhancing generalization. Significantly, models incorporating SGDrop display more expansive attributions and neural activity, offering a more comprehensive view of input images in contrast to their traditionally trained counterparts.

Title: Optical Lens Attack on Deep Learning Based Monocular Depth Estimation

• Authors: Ce Zhou (1), Qiben Yan (1), Daniel Kent (1), Guangjing Wang (1), Ziqi Zhang (2), Hayder Radha (1) ((1) Michigan State University, (2) Peking University)
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17376
• Pdf URL: https://arxiv.org/pdf/2409.17376
• Copy Paste: [[2409.17376]] Optical Lens Attack on Deep Learning Based Monocular Depth Estimation(https://arxiv.org/abs/2409.17376)
• Keywords: security, attack
• Abstract: Monocular Depth Estimation (MDE) plays a crucial role in vision-based Autonomous Driving (AD) systems. It utilizes a single-camera image to determine the depth of objects, facilitating driving decisions such as braking a few meters in front of a detected obstacle or changing lanes to avoid collision. In this paper, we investigate the security risks associated with monocular vision-based depth estimation algorithms utilized by AD systems. By exploiting the vulnerabilities of MDE and the principles of optical lenses, we introduce LensAttack, a physical attack that involves strategically placing optical lenses on the camera of an autonomous vehicle to manipulate the perceived object depths. LensAttack encompasses two attack formats: concave lens attack and convex lens attack, each utilizing different optical lenses to induce false depth perception. We begin by constructing a mathematical model of our attack, incorporating various attack parameters. Subsequently, we simulate the attack and evaluate its real-world performance in driving scenarios to demonstrate its effect on state-of-the-art MDE models. The results highlight the significant impact of LensAttack on the accuracy of depth estimation in AD systems.

Title: Beyond Redundancy: Information-aware Unsupervised Multiplex Graph Structure Learning

• Authors: Zhixiang Shen, Shuo Wang, Zhao Kang
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.17386
• Pdf URL: https://arxiv.org/pdf/2409.17386
• Copy Paste: [[2409.17386]] Beyond Redundancy: Information-aware Unsupervised Multiplex Graph Structure Learning(https://arxiv.org/abs/2409.17386)
• Keywords: robust
• Abstract: Unsupervised Multiplex Graph Learning (UMGL) aims to learn node representations on various edge types without manual labeling. However, existing research overlooks a key factor: the reliability of the graph structure. Real-world data often exhibit a complex nature and contain abundant task-irrelevant noise, severely compromising UMGL's performance. Moreover, existing methods primarily rely on contrastive learning to maximize mutual information across different graphs, limiting them to multiplex graph redundant scenarios and failing to capture view-unique task-relevant information. In this paper, we focus on a more realistic and challenging task: to unsupervisedly learn a fused graph from multiple graphs that preserve sufficient task-relevant information while removing task-irrelevant noise. Specifically, our proposed Information-aware Unsupervised Multiplex Graph Fusion framework (InfoMGF) uses graph structure refinement to eliminate irrelevant noise and simultaneously maximizes view-shared and view-unique task-relevant information, thereby tackling the frontier of non-redundant multiplex graph. Theoretical analyses further guarantee the effectiveness of InfoMGF. Comprehensive experiments against various baselines on different downstream tasks demonstrate its superior performance and robustness. Surprisingly, our unsupervised method even beats the sophisticated supervised approaches. The source code and datasets are available at this https URL.

Title: Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia

• Authors: Zhejian Zhou, Jiayu Wang, Dahua Lin, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17391
• Pdf URL: https://arxiv.org/pdf/2409.17391
• Copy Paste: [[2409.17391]] Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia(https://arxiv.org/abs/2409.17391)
• Keywords: transformer, large language model
• Abstract: Though Large Language Models (LLMs) have shown remarkable abilities in mathematics reasoning, they are still struggling with performing numeric operations accurately, such as addition and multiplication. Numbers can be tokenized into tokens in various ways by different LLMs and affect the numeric operations performance. Currently, there are two representatives: 1) Tokenize into $1$-digit, and 2) Tokenize into $1\sim 3$ digit. The difference is roughly equivalent to using different numeral systems (namely base $10$ or base $10^{3}$). In light of this, we study the scaling behavior of different numeral systems in the context of transformer-based large language models. We empirically show that a base $10$ system is consistently more data-efficient than a base $10^{2}$ or $10^{3}$ system across training data scale, model sizes under from-scratch training settings, while different number systems have very similar fine-tuning performances. We attribute this to higher token frequencies of a base $10$ system. Additionally, we reveal extrapolation behavior patterns on addition and multiplication. We identify that base $100$ and base $1000$ systems struggle on token-level discernment and token-level operations. We also sheds light on the mechanism learnt by the models.

Title: Trading through Earnings Seasons using Self-Supervised Contrastive Representation Learning

• Authors: Zhengxin Joseph Ye, Bjoern Schuller
• Subjects: cs.LG, q-fin.TR
• Abstract URL: https://arxiv.org/abs/2409.17392
• Pdf URL: https://arxiv.org/pdf/2409.17392
• Copy Paste: [[2409.17392]] Trading through Earnings Seasons using Self-Supervised Contrastive Representation Learning(https://arxiv.org/abs/2409.17392)
• Keywords: transformer
• Abstract: Earnings release is a key economic event in the financial markets and crucial for predicting stock movements. Earnings data gives a glimpse into how a company is doing financially and can hint at where its stock might go next. However, the irregularity of its release cycle makes it a challenge to incorporate this data in a medium-frequency algorithmic trading model and the usefulness of this data fades fast after it is released, making it tough for models to stay accurate over time. Addressing this challenge, we introduce the Contrastive Earnings Transformer (CET) model, a self-supervised learning approach rooted in Contrastive Predictive Coding (CPC), aiming to optimise the utilisation of earnings data. To ascertain its effectiveness, we conduct a comparative study of CET against benchmark models across diverse sectors. Our research delves deep into the intricacies of stock data, evaluating how various models, and notably CET, handle the rapidly changing relevance of earnings data over time and over different sectors. The research outcomes shed light on CET's distinct advantage in extrapolating the inherent value of earnings data over time. Its foundation on CPC allows for a nuanced understanding, facilitating consistent stock predictions even as the earnings data ages. This finding about CET presents a fresh approach to better use earnings data in algorithmic trading for predicting stock price trends.

Title: Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset

• Authors: Konstantinos Skianis, John Pavlopoulos, A. Seza Doğruöz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17397
• Pdf URL: https://arxiv.org/pdf/2409.17397
• Copy Paste: [[2409.17397]] Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset(https://arxiv.org/abs/2409.17397)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are increasingly integrated into various medical fields, including mental health support systems. However, there is a gap in research regarding the effectiveness of LLMs in non-English mental health support applications. To address this problem, we present a novel multilingual adaptation of widely-used mental health datasets, translated from English into six languages (Greek, Turkish, French, Portuguese, German, and Finnish). This dataset enables a comprehensive evaluation of LLM performance in detecting mental health conditions and assessing their severity across multiple languages. By experimenting with GPT and Llama, we observe considerable variability in performance across languages, despite being evaluated on the same translated dataset. This inconsistency underscores the complexities inherent in multilingual mental health support, where language-specific nuances and mental health data coverage can affect the accuracy of the models. Through comprehensive error analysis, we emphasize the risks of relying exclusively on large language models (LLMs) in medical settings (e.g., their potential to contribute to misdiagnoses). Moreover, our proposed approach offers significant cost savings for multilingual tasks, presenting a major advantage for broad-scale implementation.

Title: AgRegNet: A Deep Regression Network for Flower and Fruit Density Estimation, Localization, and Counting in Orchards

• Authors: Uddhav Bhattarai, Santosh Bhusal, Qin Zhang, Manoj Karkee
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17400
• Pdf URL: https://arxiv.org/pdf/2409.17400
• Copy Paste: [[2409.17400]] AgRegNet: A Deep Regression Network for Flower and Fruit Density Estimation, Localization, and Counting in Orchards(https://arxiv.org/abs/2409.17400)
• Keywords: segmentation
• Abstract: One of the major challenges for the agricultural industry today is the uncertainty in manual labor availability and the associated cost. Automated flower and fruit density estimation, localization, and counting could help streamline harvesting, yield estimation, and crop-load management strategies such as flower and fruitlet thinning. This article proposes a deep regression-based network, AgRegNet, to estimate density, count, and location of flower and fruit in tree fruit canopies without explicit object detection or polygon annotation. Inspired by popular U-Net architecture, AgRegNet is a U-shaped network with an encoder-to-decoder skip connection and modified ConvNeXt-T as an encoder feature extractor. AgRegNet can be trained based on information from point annotation and leverages segmentation information and attention modules (spatial and channel) to highlight relevant flower and fruit features while suppressing non-relevant background features. Experimental evaluation in apple flower and fruit canopy images under an unstructured orchard environment showed that AgRegNet achieved promising accuracy as measured by Structural Similarity Index (SSIM), percentage Mean Absolute Error (pMAE) and mean Average Precision (mAP) to estimate flower and fruit density, count, and centroid location, respectively. Specifically, the SSIM, pMAE, and mAP values for flower images were 0.938, 13.7%, and 0.81, respectively. For fruit images, the corresponding values were 0.910, 5.6%, and 0.93. Since the proposed approach relies on information from point annotation, it is suitable for sparsely and densely located objects. This simplified technique will be highly applicable for growers to accurately estimate yields and decide on optimal chemical and mechanical flower thinning practices.

Title: Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

• Authors: Qining Zhang, Lei Ying
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.17401
• Pdf URL: https://arxiv.org/pdf/2409.17401
• Copy Paste: [[2409.17401]] Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference(https://arxiv.org/abs/2409.17401)
• Keywords: large language model
• Abstract: Reward inference (learning a reward model from human preferences) is a critical intermediate step in Reinforcement Learning from Human Feedback (RLHF) for fine-tuning Large Language Models (LLMs) such as ChatGPT. In practice, reward inference faces several fundamental challenges, including double problem misspecification, reward model evaluation without ground truth, distribution shift, and overfitting in joint reward model and policy training. An alternative approach that avoids these pitfalls is direct policy optimization without reward inference, such as Direct Preference Optimization (DPO), which provides a much simpler pipeline and has shown empirical success in LLMs. However, DPO utilizes the closed-form expression between the optimal policy and the reward function, which only works under the bandit setting or deterministic MDPs. This paper develops two RLHF algorithms without reward inference, which work for general RL problems beyond bandits and deterministic MDPs, and general preference models beyond the Bradely-Terry model. The key idea is to estimate the local value function difference from human preferences and then approximate the policy gradient with a zeroth-order gradient approximator. For both algorithms, we establish rates of convergence in terms of the number of policy gradient iterations, as well as the number of trajectory samples and human preference queries per iteration. Our results show there exist provably efficient methods to solve general RLHF problems without reward inference.

Title: Transient Adversarial 3D Projection Attacks on Object Detection in Autonomous Driving

• Authors: Ce Zhou, Qiben Yan, Sijia Liu
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17403
• Pdf URL: https://arxiv.org/pdf/2409.17403
• Copy Paste: [[2409.17403]] Transient Adversarial 3D Projection Attacks on Object Detection in Autonomous Driving(https://arxiv.org/abs/2409.17403)
• Keywords: attack
• Abstract: Object detection is a crucial task in autonomous driving. While existing research has proposed various attacks on object detection, such as those using adversarial patches or stickers, the exploration of projection attacks on 3D surfaces remains largely unexplored. Compared to adversarial patches or stickers, which have fixed adversarial patterns, projection attacks allow for transient modifications to these patterns, enabling a more flexible attack. In this paper, we introduce an adversarial 3D projection attack specifically targeting object detection in autonomous driving scenarios. We frame the attack formulation as an optimization problem, utilizing a combination of color mapping and geometric transformation models. Our results demonstrate the effectiveness of the proposed attack in deceiving YOLOv3 and Mask R-CNN in physical settings. Evaluations conducted in an indoor environment show an attack success rate of up to 100% under low ambient light conditions, highlighting the potential damage of our attack in real-world driving scenarios.

Title: From Deception to Detection: The Dual Roles of Large Language Models in Fake News

• Authors: Dorsaf Sallami, Yuan-Chen Chang, Esma Aïmeur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17416
• Pdf URL: https://arxiv.org/pdf/2409.17416
• Copy Paste: [[2409.17416]] From Deception to Detection: The Dual Roles of Large Language Models in Fake News(https://arxiv.org/abs/2409.17416)
• Keywords: robust, large language model
• Abstract: Fake news poses a significant threat to the integrity of information ecosystems and public trust. The advent of Large Language Models (LLMs) holds considerable promise for transforming the battle against fake news. Generally, LLMs represent a double-edged sword in this struggle. One major concern is that LLMs can be readily used to craft and disseminate misleading information on a large scale. This raises the pressing questions: Can LLMs easily generate biased fake news? Do all LLMs have this capability? Conversely, LLMs offer valuable prospects for countering fake news, thanks to their extensive knowledge of the world and robust reasoning capabilities. This leads to other critical inquiries: Can we use LLMs to detect fake news, and do they outperform typical detection models? In this paper, we aim to address these pivotal questions by exploring the performance of various LLMs. Our objective is to explore the capability of various LLMs in effectively combating fake news, marking this as the first investigation to analyze seven such models. Our results reveal that while some models adhere strictly to safety protocols, refusing to generate biased or misleading content, other models can readily produce fake news across a spectrum of biases. Additionally, our results show that larger models generally exhibit superior detection abilities and that LLM-generated fake news are less likely to be detected than human-written ones. Finally, our findings demonstrate that users can benefit from LLM-generated explanations in identifying fake news.

Title: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction

• Authors: Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17422
• Pdf URL: https://arxiv.org/pdf/2409.17422
• Copy Paste: [[2409.17422]] Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction(https://arxiv.org/abs/2409.17422)
• Keywords: interpretability, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. Our research introduces a novel approach for the long context bottleneck to accelerate LLM inference and reduce GPU memory consumption. Our research demonstrates that LLMs can identify relevant tokens in the early layers before generating answers to a query. Leveraging this insight, we propose an algorithm that uses early layers of an LLM as filters to select and compress input tokens, significantly reducing the context length for subsequent processing. Our method, GemFilter, demonstrates substantial improvements in both speed and memory efficiency compared to existing techniques, such as standard attention and SnapKV/H2O. Notably, it achieves a 2.4$\times$ speedup and 30\% reduction in GPU memory usage compared to SOTA methods. Evaluation on the Needle in a Haystack task shows that GemFilter significantly outperforms standard attention, SnapKV and demonstrates comparable performance on the LongBench challenge. GemFilter is simple, training-free, and broadly applicable across different LLMs. Crucially, it provides interpretability by allowing humans to inspect the selected input sequence. These findings not only offer practical benefits for LLM deployment, but also enhance our understanding of LLM internal mechanisms, paving the way for further optimizations in LLM design and inference. Our code is available at \url{this https URL}.

Title: HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows

• Authors: Wenlin Yao, Haitao Mi, Dong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17433
• Pdf URL: https://arxiv.org/pdf/2409.17433
• Copy Paste: [[2409.17433]] HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows(https://arxiv.org/abs/2409.17433)
• Keywords: large language model
• Abstract: Despite recent advancements in large language models (LLMs), their performance on complex reasoning problems requiring multi-step thinking and combining various skills is still limited. To address this, we propose a novel framework HDFlow for complex reasoning with LLMs that combines fast and slow thinking modes in an adaptive manner. Our approach consists of two key components: 1) a new approach for slow, deliberate reasoning called Dynamic Workflow, which automatically decomposes complex problems into more manageable sub-tasks and dynamically designs a workflow to assemble specialized LLM or symbolic reasoning tools to solve sub-tasks; 2) Hybrid Thinking, a general framework that dynamically combines fast and slow thinking based on problem complexity. Finally, we propose an easy-to-scale method for automatically synthesizing a large-scale dataset of 27K challenging reasoning problems for complex reasoning and a hybrid thinking tuning method that trains smaller LLMs on this dataset to internalize the fast/slow hybrid reasoning strategies. Experiments on four reasoning benchmark datasets demonstrate that our slow thinking with dynamic workflows significantly outperforms Chain-of-Thought, and hybrid thinking achieves the highest accuracy while providing an effective balance between computational efficiency and performance. Fine-tuning using our hybrid thinking approach also significantly boosts the complex reasoning capabilities of open-source language models. The results showcase the promise of slow thinking, dynamic workflows, and hybrid thinking in expanding the frontier of complex problem-solving with LLMs\footnote{Code and data will be released at \url{this https URL}.}.

Title: Rejection Sampling IMLE: Designing Priors for Better Few-Shot Image Synthesis

• Authors: Chirag Vashist, Shichong Peng, Ke Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17439
• Pdf URL: https://arxiv.org/pdf/2409.17439
• Copy Paste: [[2409.17439]] Rejection Sampling IMLE: Designing Priors for Better Few-Shot Image Synthesis(https://arxiv.org/abs/2409.17439)
• Keywords: diffusion, generative
• Abstract: An emerging area of research aims to learn deep generative models with limited training data. Prior generative models like GANs and diffusion models require a lot of data to perform well, and their performance degrades when they are trained on only a small amount of data. A recent technique called Implicit Maximum Likelihood Estimation (IMLE) has been adapted to the few-shot setting, achieving state-of-the-art performance. However, current IMLE-based approaches encounter challenges due to inadequate correspondence between the latent codes selected for training and those drawn during inference. This results in suboptimal test-time performance. We theoretically show a way to address this issue and propose RS-IMLE, a novel approach that changes the prior distribution used for training. This leads to substantially higher quality image generation compared to existing GAN and IMLE-based methods, as validated by comprehensive experiments conducted on nine few-shot image datasets.

Title: Enhancing Financial Sentiment Analysis with Expert-Designed Hint

• Authors: Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17448
• Pdf URL: https://arxiv.org/pdf/2409.17448
• Copy Paste: [[2409.17448]] Enhancing Financial Sentiment Analysis with Expert-Designed Hint(https://arxiv.org/abs/2409.17448)
• Keywords: large language model
• Abstract: This paper investigates the role of expert-designed hint in enhancing sentiment analysis on financial social media posts. We explore the capability of large language models (LLMs) to empathize with writer perspectives and analyze sentiments. Our findings reveal that expert-designed hint, i.e., pointing out the importance of numbers, significantly improve performances across various LLMs, particularly in cases requiring perspective-taking skills. Further analysis on tweets containing different types of numerical data demonstrates that the inclusion of expert-designed hint leads to notable improvements in sentiment analysis performance, especially for tweets with monetary-related numbers. Our findings contribute to the ongoing discussion on the applicability of Theory of Mind in NLP and open new avenues for improving sentiment analysis in financial domains through the strategic use of expert knowledge.

Title: AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing

• Authors: Hanbo Bi, Yingchao Feng, Yongqiang Mao, Jianning Pei, Wenhui Diao, Hongqi Wang, Xian Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17453
• Pdf URL: https://arxiv.org/pdf/2409.17453
• Copy Paste: [[2409.17453]] AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing(https://arxiv.org/abs/2409.17453)
• Keywords: transformer, segmentation
• Abstract: Few-shot Segmentation (FSS) aims to segment the interested objects in the query image with just a handful of labeled samples (i.e., support images). Previous schemes would leverage the similarity between support-query pixel pairs to construct the pixel-level semantic correlation. However, in remote sensing scenarios with extreme intra-class variations and cluttered backgrounds, such pixel-level correlations may produce tremendous mismatches, resulting in semantic ambiguity between the query foreground (FG) and background (BG) pixels. To tackle this problem, we propose a novel Agent Mining Transformer (AgMTR), which adaptively mines a set of local-aware agents to construct agent-level semantic correlation. Compared with pixel-level semantics, the given agents are equipped with local-contextual information and possess a broader receptive field. At this point, different query pixels can selectively aggregate the fine-grained local semantics of different agents, thereby enhancing the semantic clarity between query FG and BG pixels. Concretely, the Agent Learning Encoder (ALE) is first proposed to erect the optimal transport plan that arranges different agents to aggregate support semantics under different local regions. Then, for further optimizing the agents, the Agent Aggregation Decoder (AAD) and the Semantic Alignment Decoder (SAD) are constructed to break through the limited support set for mining valuable class-specific semantics from unlabeled data sources and the query image itself, respectively. Extensive experiments on the remote sensing benchmark iSAID indicate that the proposed method achieves state-of-the-art performance. Surprisingly, our method remains quite competitive when extended to more common natural scenarios, i.e., PASCAL-5i and COCO-20i.

Title: Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models

• Authors: Yuqing Zhou, Ruixiang Tang, Ziyu Yao, Ziwei Zhu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17455
• Pdf URL: https://arxiv.org/pdf/2409.17455
• Copy Paste: [[2409.17455]] Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models(https://arxiv.org/abs/2409.17455)
• Keywords: robust, large language model
• Abstract: Language models (LMs), despite their advances, often depend on spurious correlations, undermining their accuracy and generalizability. This study addresses the overlooked impact of subtler, more complex shortcuts that compromise model reliability beyond oversimplified shortcuts. We introduce a comprehensive benchmark that categorizes shortcuts into occurrence, style, and concept, aiming to explore the nuanced ways in which these shortcuts influence the performance of LMs. Through extensive experiments across traditional LMs, large language models, and state-of-the-art robust models, our research systematically investigates models' resilience and susceptibilities to sophisticated shortcuts. Our benchmark and code can be found at: this https URL.

Title: CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches

• Authors: Sifan Wu, Amir Khasahmadi, Mor Katz, Pradeep Kumar Jayaraman, Yewen Pu, Karl Willis, Bang Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17457
• Pdf URL: https://arxiv.org/pdf/2409.17457
• Copy Paste: [[2409.17457]] CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches(https://arxiv.org/abs/2409.17457)
• Keywords: generative, large language model
• Abstract: Parametric Computer-Aided Design (CAD) is central to contemporary mechanical design. However, it encounters challenges in achieving precise parametric sketch modeling and lacks practical evaluation metrics suitable for mechanical design. We harness the capabilities of pre-trained foundation models, renowned for their successes in natural language processing and computer vision, to develop generative models specifically for CAD. These models are adept at understanding complex geometries and design reasoning, a crucial advancement in CAD technology. In this paper, we propose CadVLM, an end-to-end vision language model for CAD generation. Our approach involves adapting pre-trained foundation models to manipulate engineering sketches effectively, integrating both sketch primitive sequences and sketch images. Extensive experiments demonstrate superior performance on multiple CAD sketch generation tasks such as CAD autocompletion, CAD autoconstraint, and image conditional generation. To our knowledge, this is the first instance of a multimodal Large Language Model (LLM) being successfully applied to parametric CAD generation, representing a pioneering step in the field of computer-aided mechanical design.

Title: RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking

• Authors: Yifan Jiang, Kriti Aggarwal, Tanmay Laud, Kashif Munir, Jay Pujara, Subhabrata Mukherjee
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17458
• Pdf URL: https://arxiv.org/pdf/2409.17458
• Copy Paste: [[2409.17458]] RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking(https://arxiv.org/abs/2409.17458)
• Keywords: security, attack, large language model
• Abstract: The rapid progress of Large Language Models (LLMs) has opened up new opportunities across various domains and applications; yet it also presents challenges related to potential misuse. To mitigate such risks, red teaming has been employed as a proactive security measure to probe language models for harmful outputs via jailbreak attacks. However, current jailbreak attack approaches are single-turn with explicit malicious queries that do not fully capture the complexity of real-world interactions. In reality, users can engage in multi-turn interactions with LLM-based chat assistants, allowing them to conceal their true intentions in a more covert manner. To bridge this gap, we, first, propose a new jailbreak approach, RED QUEEN ATTACK. This method constructs a multi-turn scenario, concealing the malicious intent under the guise of preventing harm. We craft 40 scenarios that vary in turns and select 14 harmful categories to generate 56k multi-turn attack data points. We conduct comprehensive experiments on the RED QUEEN ATTACK with four representative LLM families of different sizes. Our experiments reveal that all LLMs are vulnerable to RED QUEEN ATTACK, reaching 87.62% attack success rate on GPT-4o and 75.4% on Llama3-70B. Further analysis reveals that larger models are more susceptible to the RED QUEEN ATTACK, with multi-turn structures and concealment strategies contributing to its success. To prioritize safety, we introduce a straightforward mitigation strategy called RED QUEEN GUARD, which aligns LLMs to effectively counter adversarial attacks. This approach reduces the attack success rate to below 1% while maintaining the model's performance across standard benchmarks. Full implementation and dataset are publicly accessible at this https URL.

Title: Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards

• Authors: Heejin Do, Sangwon Ryu, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17472
• Pdf URL: https://arxiv.org/pdf/2409.17472
• Copy Paste: [[2409.17472]] Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards(https://arxiv.org/abs/2409.17472)
• Keywords: robust
• Abstract: Recent advances in automated essay scoring (AES) have shifted towards evaluating multiple traits to provide enriched feedback. Like typical AES systems, multi-trait AES employs the quadratic weighted kappa (QWK) to measure agreement with human raters, aligning closely with the rating schema; however, its non-differentiable nature prevents its direct use in neural network training. In this paper, we propose Scoring-aware Multi-reward Reinforcement Learning (SaMRL), which integrates actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES. Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation, as RL requires probability distributions; instead, we adopt an autoregressive score generation framework to leverage token generation probabilities for robust multi-trait score predictions. Empirical analyses demonstrate that SaMRL facilitates model training, notably enhancing scoring of previously inferior prompts.

Title: Learning Quantized Adaptive Conditions for Diffusion Models

• Authors: Yuchen Liang, Yuchuan Tian, Lei Yu, Huao Tang, Jie Hu, Xiangzhong Fang, Hanting Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17487
• Pdf URL: https://arxiv.org/pdf/2409.17487
• Copy Paste: [[2409.17487]] Learning Quantized Adaptive Conditions for Diffusion Models(https://arxiv.org/abs/2409.17487)
• Keywords: diffusion
• Abstract: The curvature of ODE trajectories in diffusion models hinders their ability to generate high-quality images in a few number of function evaluations (NFE). In this paper, we propose a novel and effective approach to reduce trajectory curvature by utilizing adaptive conditions. By employing a extremely light-weight quantized encoder, our method incurs only an additional 1% of training parameters, eliminates the need for extra regularization terms, yet achieves significantly better sample quality. Our approach accelerates ODE sampling while preserving the downstream task image editing capabilities of SDE techniques. Extensive experiments verify that our method can generate high quality results under extremely limited sampling costs. With only 6 NFE, we achieve 5.14 FID on CIFAR-10, 6.91 FID on FFHQ 64x64 and 3.10 FID on AFHQv2.

Title: Does Worst-Performing Agent Lead the Pack? Analyzing Agent Dynamics in Unified Distributed SGD

• Authors: Jie Hu, Yi-Ting Ma, Do Young Eun
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.17499
• Pdf URL: https://arxiv.org/pdf/2409.17499
• Copy Paste: [[2409.17499]] Does Worst-Performing Agent Lead the Pack? Analyzing Agent Dynamics in Unified Distributed SGD(https://arxiv.org/abs/2409.17499)
• Keywords: privacy, federate
• Abstract: Distributed learning is essential to train machine learning algorithms across heterogeneous agents while maintaining data privacy. We conduct an asymptotic analysis of Unified Distributed SGD (UD-SGD), exploring a variety of communication patterns, including decentralized SGD and local SGD within Federated Learning (FL), as well as the increasing communication interval in the FL setting. In this study, we assess how different sampling strategies, such as i.i.d. sampling, shuffling, and Markovian sampling, affect the convergence speed of UD-SGD by considering the impact of agent dynamics on the limiting covariance matrix as described in the Central Limit Theorem (CLT). Our findings not only support existing theories on linear speedup and asymptotic network independence, but also theoretically and empirically show how efficient sampling strategies employed by individual agents contribute to overall convergence in UD-SGD. Simulations reveal that a few agents using highly efficient sampling can achieve or surpass the performance of the majority employing moderately improved strategies, providing new insights beyond traditional analyses focusing on the worst-performing agent.

Title: HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection

• Authors: Xuefeng Du, Chaowei Xiao, Yixuan Li
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17504
• Pdf URL: https://arxiv.org/pdf/2409.17504
• Copy Paste: [[2409.17504]] HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection(https://arxiv.org/abs/2409.17504)
• Keywords: large language model
• Abstract: The surge in applications of large language models (LLMs) has prompted concerns about the generation of misleading or fabricated information, known as hallucinations. Therefore, detecting hallucinations has become critical to maintaining trust in LLM-generated content. A primary challenge in learning a truthfulness classifier is the lack of a large amount of labeled truthful and hallucinated data. To address the challenge, we introduce HaloScope, a novel learning framework that leverages the unlabeled LLM generations in the wild for hallucination detection. Such unlabeled data arises freely upon deploying LLMs in the open world, and consists of both truthful and hallucinated information. To harness the unlabeled data, we present an automated membership estimation score for distinguishing between truthful and untruthful generations within unlabeled mixture data, thereby enabling the training of a binary truthfulness classifier on top. Importantly, our framework does not require extra data collection and human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiments show that HaloScope can achieve superior hallucination detection performance, outperforming the competitive rivals by a significant margin. Code is available at this https URL.

Title: Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE

• Authors: Xun Zhu, Ying Hu, Fanbin Mo, Miao Li, Ji Wu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17508
• Pdf URL: https://arxiv.org/pdf/2409.17508
• Copy Paste: [[2409.17508]] Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE(https://arxiv.org/abs/2409.17508)
• Keywords: extraction, large language model
• Abstract: Multi-modal large language models (MLLMs) have shown impressive capabilities as a general-purpose interface for various visual and linguistic tasks. However, building a unified MLLM for multi-task learning in the medical field remains a thorny challenge. To mitigate the tug-of-war problem of multi-modal multi-task optimization, recent advances primarily focus on improving the LLM components, while neglecting the connector that bridges the gap between modalities. In this paper, we introduce Uni-Med, a novel medical generalist foundation model which consists of a universal visual feature extraction module, a connector mixture-of-experts (CMoE) module, and an LLM. Benefiting from the proposed CMoE that leverages a well-designed router with a mixture of projection experts at the connector, Uni-Med achieves efficient solution to the tug-of-war problem and can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprehension, referring expression generation and image classification. To the best of our knowledge, Uni-Med is the first effort to tackle multi-task interference at the connector. Extensive ablation experiments validate the effectiveness of introducing CMoE under any configuration, with up to an average 8% performance gains. We further provide interpretation analysis of the tug-of-war problem from the perspective of gradient optimization and parameter statistics. Compared to previous state-of-the-art medical MLLMs, Uni-Med achieves competitive or superior evaluation metrics on diverse tasks. Code, data and model will be soon available at GitHub.

Title: BioZero: An Efficient and Privacy-Preserving Decentralized Biometric Authentication Protocol on Open Blockchain

• Authors: Junhao Lai, Taotao Wang, Shengli Zhang, Qing Yang, Soung Chang Liew
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17509
• Pdf URL: https://arxiv.org/pdf/2409.17509
• Copy Paste: [[2409.17509]] BioZero: An Efficient and Privacy-Preserving Decentralized Biometric Authentication Protocol on Open Blockchain(https://arxiv.org/abs/2409.17509)
• Keywords: secure, security, privacy, protect, attack, biometric
• Abstract: Digital identity plays a vital role in enabling secure access to resources and services in the digital world. Traditional identity authentication methods, such as password-based and biometric authentications, have limitations in terms of security, privacy, and scalability. Decentralized authentication approaches leveraging blockchain technology have emerged as a promising solution. However, existing decentralized authentication methods often rely on indirect identity verification (e.g. using passwords or digital signatures as authentication credentials) and face challenges such as Sybil attacks. In this paper, we propose BioZero, an efficient and privacy-preserving decentralized biometric authentication protocol that can be implemented on open blockchain. BioZero leverages Pedersen commitment and homomorphic computation to protect user biometric privacy while enabling efficient verification. We enhance the protocol with non-interactive homomorphic computation and employ zero-knowledge proofs for secure on-chain verification. The unique aspect of BioZero is that it is fully decentralized and can be executed by blockchain smart contracts in a very efficient way. We analyze the security of BioZero and validate its performance through a prototype implementation. The results demonstrate the effectiveness, efficiency, and security of BioZero in decentralized authentication scenarios. Our work contributes to the advancement of decentralized identity authentication using biometrics.

Title: Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted Code

• Authors: Gary A. McCully, John D. Hastings, Shengjie Xu, Adam Fortier
• Subjects: cs.CR, cs.CL, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.17513
• Pdf URL: https://arxiv.org/pdf/2409.17513
• Copy Paste: [[2409.17513]] Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted Code(https://arxiv.org/abs/2409.17513)
• Keywords: security, defense, transformer
• Abstract: Ransomware and other forms of malware cause significant financial and operational damage to organizations by exploiting long-standing and often difficult-to-detect software vulnerabilities. To detect vulnerabilities such as buffer overflows in compiled code, this research investigates the application of unidirectional transformer-based embeddings, specifically GPT-2. Using a dataset of LLVM functions, we trained a GPT-2 model to generate embeddings, which were subsequently used to build LSTM neural networks to differentiate between vulnerable and non-vulnerable code. Our study reveals that embeddings from the GPT-2 model significantly outperform those from bidirectional models of BERT and RoBERTa, achieving an accuracy of 92.5% and an F1-score of 89.7%. LSTM neural networks were developed with both frozen and unfrozen embedding model layers. The model with the highest performance was achieved when the embedding layers were unfrozen. Further, the research finds that, in exploring the impact of different optimizers within this domain, the SGD optimizer demonstrates superior performance over Adam. Overall, these findings reveal important insights into the potential of unidirectional transformer-based approaches in enhancing cybersecurity defenses.

Title: Dataset Distillation-based Hybrid Federated Learning on Non-IID Data

• Authors: Xiufang Shi, Wei Zhang, Mincheng Wu, Guangyi Liu, Zhenyu Wen, Shibo He, Tejal Shah, Rajiv Ranjan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17517
• Pdf URL: https://arxiv.org/pdf/2409.17517
• Copy Paste: [[2409.17517]] Dataset Distillation-based Hybrid Federated Learning on Non-IID Data(https://arxiv.org/abs/2409.17517)
• Keywords: federate
• Abstract: In federated learning, the heterogeneity of client data has a great impact on the performance of model training. Many heterogeneity issues in this process are raised by non-independently and identically distributed (Non-IID) data. This study focuses on the issue of label distribution skew. To address it, we propose a hybrid federated learning framework called HFLDD, which integrates dataset distillation to generate approximately independent and equally distributed (IID) data, thereby improving the performance of model training. Particularly, we partition the clients into heterogeneous clusters, where the data labels among different clients within a cluster are unbalanced while the data labels among different clusters are balanced. The cluster headers collect distilled data from the corresponding cluster members, and conduct model training in collaboration with the server. This training process is like traditional federated learning on IID data, and hence effectively alleviates the impact of Non-IID data on model training. Furthermore, we compare our proposed method with typical baseline methods on public datasets. Experimental results demonstrate that when the data labels are severely imbalanced, the proposed HFLDD outperforms the baseline methods in terms of both test accuracy and communication cost.

Title: Multi-Designated Detector Watermarking for Language Models

• Authors: Zhengan Huang, Gongxian Zeng, Xin Mu, Yu Wang, Yue Yu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17518
• Pdf URL: https://arxiv.org/pdf/2409.17518
• Copy Paste: [[2409.17518]] Multi-Designated Detector Watermarking for Language Models(https://arxiv.org/abs/2409.17518)
• Keywords: security, watermark, large language model
• Abstract: In this paper, we initiate the study of \emph{multi-designated detector watermarking (MDDW)} for large language models (LLMs). This technique allows model providers to generate watermarked outputs from LLMs with two key properties: (i) only specific, possibly multiple, designated detectors can identify the watermarks, and (ii) there is no perceptible degradation in the output quality for ordinary users. We formalize the security definitions for MDDW and present a framework for constructing MDDW for any LLM using multi-designated verifier signatures (MDVS). Recognizing the significant economic value of LLM outputs, we introduce claimability as an optional security feature for MDDW, enabling model providers to assert ownership of LLM outputs within designated-detector settings. To support claimable MDDW, we propose a generic transformation converting any MDVS to a claimable MDVS. Our implementation of the MDDW scheme highlights its advanced functionalities and flexibility over existing methods, with satisfactory performance metrics.

Title: EAGLE: Egocentric AGgregated Language-video Engine

• Authors: Jing Bi, Yunlong Tang, Luchuan Song, Ali Vosoughi, Nguyen Nguyen, Chenliang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17523
• Pdf URL: https://arxiv.org/pdf/2409.17523
• Copy Paste: [[2409.17523]] EAGLE: Egocentric AGgregated Language-video Engine(https://arxiv.org/abs/2409.17523)
• Keywords: large language model
• Abstract: The rapid evolution of egocentric video analysis brings new insights into understanding human activities and intentions from a first-person perspective. Despite this progress, the fragmentation in tasks like action recognition, procedure learning, and moment retrieval, \etc, coupled with inconsistent annotations and isolated model development, hinders a holistic interpretation of video content. In response, we introduce the EAGLE (Egocentric AGgregated Language-video Engine) model and the EAGLE-400K dataset to provide a unified framework that integrates various egocentric video understanding tasks. EAGLE-400K, the \textit{first} large-scale instruction-tuning dataset tailored for egocentric video, features 400K diverse samples to enhance a broad spectrum of tasks from activity recognition to procedure knowledge learning. Moreover, EAGLE, a strong video multimodal large language model (MLLM), is designed to effectively capture both spatial and temporal information. In addition, we propose a set of evaluation metrics designed to facilitate a thorough assessment of MLLM for egocentric video understanding. Our extensive experiments demonstrate EAGLE's superior performance over existing models, highlighting its ability to balance task-specific understanding with holistic video interpretation. With EAGLE, we aim to pave the way for research opportunities and practical applications in real-world scenarios.

Title: JoyType: A Robust Design for Multilingual Visual Text Creation

• Authors: Chao Li, Chen Jiang, Xiaolong Liu, Jun Zhao, Guoxin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17524
• Pdf URL: https://arxiv.org/pdf/2409.17524
• Copy Paste: [[2409.17524]] JoyType: A Robust Design for Multilingual Visual Text Creation(https://arxiv.org/abs/2409.17524)
• Keywords: robust, diffusion
• Abstract: Generating images with accurately represented text, especially in non-Latin languages, poses a significant challenge for diffusion models. Existing approaches, such as the integration of hint condition diagrams via auxiliary networks (e.g., ControlNet), have made strides towards addressing this issue. However, diffusion models often fall short in tasks requiring controlled text generation, such as specifying particular fonts or producing text in small fonts. In this paper, we introduce a novel approach for multilingual visual text creation, named JoyType, designed to maintain the font style of text during the image generation process. Our methodology begins with assembling a training dataset, JoyType-1M, comprising 1 million pairs of data. Each pair includes an image, its description, and glyph instructions corresponding to the font style within the image. We then developed a text control network, Font ControlNet, tasked with extracting font style information to steer the image generation. To further enhance our model's ability to maintain font style, notably in generating small-font text, we incorporated a multi-layer OCR-aware loss into the diffusion process. This enhancement allows JoyType to direct text rendering using low-level descriptors. Our evaluations, based on both visual and accuracy metrics, demonstrate that JoyType significantly outperforms existing state-of-the-art methods. Additionally, JoyType can function as a plugin, facilitating the creation of varied image styles in conjunction with other stable diffusion models on HuggingFace and CivitAI. Our project is open-sourced on this https URL.

Title: Drone Stereo Vision for Radiata Pine Branch Detection and Distance Measurement: Integrating SGBM and Segmentation Models

• Authors: Yida Lin, Bing Xue, Mengjie Zhang, Sam Schofield, Richard Green
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17526
• Pdf URL: https://arxiv.org/pdf/2409.17526
• Copy Paste: [[2409.17526]] Drone Stereo Vision for Radiata Pine Branch Detection and Distance Measurement: Integrating SGBM and Segmentation Models(https://arxiv.org/abs/2409.17526)
• Keywords: segmentation
• Abstract: Manual pruning of radiata pine trees presents significant safety risks due to their substantial height and the challenging terrains in which they thrive. To address these risks, this research proposes the development of a drone-based pruning system equipped with specialized pruning tools and a stereo vision camera, enabling precise detection and trimming of branches. Deep learning algorithms, including YOLO and Mask R-CNN, are employed to ensure accurate branch detection, while the Semi-Global Matching algorithm is integrated to provide reliable distance estimation. The synergy between these techniques facilitates the precise identification of branch locations and enables efficient, targeted pruning. Experimental results demonstrate that the combined implementation of YOLO and SGBM enables the drone to accurately detect branches and measure their distances from the drone. This research not only improves the safety and efficiency of pruning operations but also makes a significant contribution to the advancement of drone technology in the automation of agricultural and forestry practices, laying a foundational framework for further innovations in environmental management.

Title: Data Proportion Detection for Optimized Data Management for Large Language Models

• Authors: Hao Liang, Keshi Zhao, Yajie Yang, Bin Cui, Guosheng Dong, Zenan Zhou, Wentao Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17527
• Pdf URL: https://arxiv.org/pdf/2409.17527
• Copy Paste: [[2409.17527]] Data Proportion Detection for Optimized Data Management for Large Language Models(https://arxiv.org/abs/2409.17527)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks and domains, with data preparation playing a critical role in achieving these results. Pre-training data typically combines information from multiple domains. To maximize performance when integrating data from various domains, determining the optimal data proportion is essential. However, state-of-the-art (SOTA) LLMs rarely disclose details about their pre-training data, making it difficult for researchers to identify ideal data proportions. In this paper, we introduce a new topic, \textit{data proportion detection}, which enables the automatic estimation of pre-training data proportions by analyzing the generated outputs of LLMs. We provide rigorous theoretical proofs, practical algorithms, and preliminary experimental results for data proportion detection. Based on these findings, we offer valuable insights into the challenges and future directions for effective data proportion detection and data management.

Title: SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion

• Authors: Ming Dai, Lingfeng Yang, Yihao Xu, Zhenhua Feng, Wankou Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17531
• Pdf URL: https://arxiv.org/pdf/2409.17531
• Copy Paste: [[2409.17531]] SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion(https://arxiv.org/abs/2409.17531)
• Keywords: robust, transformer
• Abstract: Visual grounding is a common vision task that involves grounding descriptive sentences to the corresponding regions of an image. Most existing methods use independent image-text encoding and apply complex hand-crafted modules or encoder-decoder architectures for modal interaction and query reasoning. However, their performance significantly drops when dealing with complex textual expressions. This is because the former paradigm only utilizes limited downstream data to fit the multi-modal feature fusion. Therefore, it is only effective when the textual expressions are relatively simple. In contrast, given the wide diversity of textual expressions and the uniqueness of downstream training data, the existing fusion module, which extracts multimodal content from a visual-linguistic context, has not been fully investigated. In this paper, we present a simple yet robust transformer-based framework, SimVG, for visual grounding. Specifically, we decouple visual-linguistic feature fusion from downstream tasks by leveraging existing multimodal pre-trained models and incorporating additional object tokens to facilitate deep integration of downstream and pre-training tasks. Furthermore, we design a dynamic weight-balance distillation method in the multi-branch synchronous learning process to enhance the representation capability of the simpler branch. This branch only consists of a lightweight MLP, which simplifies the structure and improves reasoning speed. Experiments on six widely used VG datasets, i.e., RefCOCO/+/g, ReferIt, Flickr30K, and GRefCOCO, demonstrate the superiority of SimVG. Finally, the proposed method not only achieves improvements in efficiency and convergence speed but also attains new state-of-the-art performance on these benchmarks. Codes and models will be available at \url{this https URL}.

Title: Privacy-Preserving Redaction of Diagnosis Data through Source Code Analysis

• Authors: Lixi Zhou, Lei Yu, Jia Zou, Hong Min
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17535
• Pdf URL: https://arxiv.org/pdf/2409.17535
• Copy Paste: [[2409.17535]] Privacy-Preserving Redaction of Diagnosis Data through Source Code Analysis(https://arxiv.org/abs/2409.17535)
• Keywords: privacy, protect
• Abstract: Protecting sensitive information in diagnostic data such as logs, is a critical concern in the industrial software diagnosis and debugging process. While there are many tools developed to automatically redact the logs for identifying and removing sensitive information, they have severe limitations which can cause either over redaction and loss of critical diagnostic information (false positives), or disclosure of sensitive information (false negatives), or both. To address the problem, in this paper, we argue for a source code analysis approach for log redaction. To identify a log message containing sensitive information, our method locates the corresponding log statement in the source code with logger code augmentation, and checks if the log statement outputs data from sensitive sources by using the data flow graph built from the source code. Appropriate redaction rules are further applied depending on the sensitiveness of the data sources to preserve the privacy information in the logs. We conducted experimental evaluation and comparison with other popular baselines. The results demonstrate that our approach can significantly improve the detection precision of the sensitive information and reduce both false positives and negatives.

Title: On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy

• Authors: Saber Malekmohammadi, Golnoosh Farnadi
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17538
• Pdf URL: https://arxiv.org/pdf/2409.17538
• Copy Paste: [[2409.17538]] On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy(https://arxiv.org/abs/2409.17538)
• Keywords: privacy, transformer
• Abstract: A significant approach in natural language processing involves large-scale pre-training on general domain data followed by adaptation to specific tasks or domains. As models grow in size, full fine-tuning all parameters becomes increasingly impractical. To address this, some methods for low-rank task adaptation of language models have been proposed, e.g. LoRA and FLoRA. These methods keep the pre-trained model weights fixed and incorporate trainable low-rank decomposition matrices into some layers of the transformer architecture, called adapters. This approach significantly reduces the number of trainable parameters required for downstream tasks compared to full fine-tuning all parameters. In this work, we look at low-rank adaptation from the lens of data privacy. We show theoretically that the low-rank adaptation used in LoRA and FLoRA is equivalent to injecting some random noise into the batch gradients w.r.t the adapter parameters coming from their full fine-tuning, and we quantify the variance of the injected noise. By establishing a Berry-Esseen type bound on the total variation distance between the noise distribution and a Gaussian distribution with the same variance, we show that the dynamics of LoRA and FLoRA are very close to differentially private full fine-tuning the adapters, which suggests that low-rank adaptation implicitly provides privacy w.r.t the fine-tuning data. Finally, using Johnson-Lindenstrauss lemma, we show that when augmented with gradient clipping, low-rank adaptation is almost equivalent to differentially private full fine-tuning adapters with a fixed noise scale.

Title: Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models

• Authors: Tongxuan Liu, Wenjiang Xu, Weizhe Huang, Xingyu Wang, Jiaxing Wang, Hailong Yang, Jing Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17539
• Pdf URL: https://arxiv.org/pdf/2409.17539
• Copy Paste: [[2409.17539]] Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models(https://arxiv.org/abs/2409.17539)
• Keywords: extraction, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Although some prompting methods, such as Chain-of-Thought, can improve the reasoning ability of LLMs to some extent, they suffer from an unfaithful issue where derived conclusions may not align with the generated reasoning chain. To address this issue, some studies employ the approach of propositional logic to further enhance logical reasoning abilities of LLMs. However, the potential omissions in the extraction of logical expressions in these methods can cause information loss in the logical reasoning process, thereby generating incorrect results. To this end, we propose Logic-of-Thought (LoT) prompting which employs propositional logic to generate expanded logical information from input context, and utilizes the generated logical information as an additional augmentation to the input prompts, thereby enhancing the capability of logical reasoning. The LoT is orthogonal to existing prompting methods and can be seamlessly integrated with them. Extensive experiments demonstrate that LoT boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. In particular, the LoT enhances Chain-of-Thought's performance on the ReClor dataset by +4.35%; moreover, it improves Chain-of-Thought with Self-Consistency's performance on LogiQA by +5%; additionally, it boosts performance of Tree-of-Thoughts on ProofWriter dataset by +8%.

Title: A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation

• Authors: Masato Ishii, Akio Hayakawa, Takashi Shibuya, Yuki Mitsufuji
• Subjects: cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.17550
• Pdf URL: https://arxiv.org/pdf/2409.17550
• Copy Paste: [[2409.17550]] A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation(https://arxiv.org/abs/2409.17550)
• Keywords: diffusion
• Abstract: In this work, we build a simple but strong baseline for sounding video generation. Given base diffusion models for audio and video, we integrate them with additional modules into a single model and train it to make the model jointly generate audio and video. To enhance alignment between audio-video pairs, we introduce two novel mechanisms in our model. The first one is timestep adjustment, which provides different timestep information to each base model. It is designed to align how samples are generated along with timesteps across modalities. The second one is a new design of the additional modules, termed Cross-Modal Conditioning as Positional Encoding (CMC-PE). In CMC-PE, cross-modal information is embedded as if it represents temporal position information, and the embeddings are fed into the model like positional encoding. Compared with the popular cross-attention mechanism, CMC-PE provides a better inductive bias for temporal alignment in the generated data. Experimental results validate the effectiveness of the two newly introduced mechanisms and also demonstrate that our method outperforms existing methods.

Title: Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking

• Authors: Pengcheng Shao, Tianyang Xu, Xuefeng Zhu, Xiaojun Wu, Josef Kittler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17560
• Pdf URL: https://arxiv.org/pdf/2409.17560
• Copy Paste: [[2409.17560]] Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking(https://arxiv.org/abs/2409.17560)
• Keywords: transformer
• Abstract: Event-based bionic camera asynchronously captures dynamic scenes with high temporal resolution and high dynamic range, offering potential for the integration of events and RGB under conditions of illumination degradation and fast motion. Existing RGB-E tracking methods model event characteristics utilising attention mechanism of Transformer before integrating both modalities. Nevertheless, these methods involve aggregating the event stream into a single event frame, lacking the utilisation of the temporal information inherent in the event this http URL, the traditional attention mechanism is well-suited for dense semantic features, while the attention mechanism for sparse event features require revolution. In this paper, we propose a dynamic event subframe splitting strategy to split the event stream into more fine-grained event clusters, aiming to capture spatio-temporal features that contain motion cues. Based on this, we design an event-based sparse attention mechanism to enhance the interaction of event features in temporal and spatial dimensions. The experimental results indicate that our method outperforms existing state-of-the-art methods on the FE240 and COESOT datasets, providing an effective processing manner for the event data.

Title: General Compression Framework for Efficient Transformer Object Tracking

• Authors: Lingyi Hong, Jinglun Li, Xinyu Zhou, Shilin Yan, Pinxue Guo, Kaixun Jiang, Zhaoyu Chen, Shuyong Gao, Wei Zhang, Hong Lu, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17564
• Pdf URL: https://arxiv.org/pdf/2409.17564
• Copy Paste: [[2409.17564]] General Compression Framework for Efficient Transformer Object Tracking(https://arxiv.org/abs/2409.17564)
• Keywords: transformer
• Abstract: Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model's ability to replicate the teacher model's behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model's compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about 96% performance on LaSOT (66.1% AUC) while achieves 2.17x speed up.

Title: Pixel-Space Post-Training of Latent Diffusion Models

• Authors: Christina Zhang, Simran Motwani, Matthew Yu, Ji Hou, Felix Juefei-Xu, Sam Tsai, Peter Vajda, Zijian He, Jialiang Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17565
• Pdf URL: https://arxiv.org/pdf/2409.17565
• Copy Paste: [[2409.17565]] Pixel-Space Post-Training of Latent Diffusion Models(https://arxiv.org/abs/2409.17565)
• Keywords: diffusion, transformer
• Abstract: Latent diffusion models (LDMs) have made significant advancements in the field of image generation in recent years. One major advantage of LDMs is their ability to operate in a compressed latent space, allowing for more efficient training and deployment. However, despite these advantages, challenges with LDMs still remain. For example, it has been observed that LDMs often generate high-frequency details and complex compositions imperfectly. We hypothesize that one reason for these flaws is due to the fact that all pre- and post-training of LDMs are done in latent space, which is typically $8 \times 8$ lower spatial-resolution than the output images. To address this issue, we propose adding pixel-space supervision in the post-training process to better preserve high-frequency details. Experimentally, we show that adding a pixel-space objective significantly improves both supervised quality fine-tuning and preference-based post-training by a large margin on a state-of-the-art DiT transformer and U-Net diffusion models in both visual quality and visual flaw metrics, while maintaining the same text alignment quality.

Title: Flexiffusion: Segment-wise Neural Architecture Search for Flexible Denoising Schedule

• Authors: Hongtao Huang, Xiaojun Chang, Lina Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17566
• Pdf URL: https://arxiv.org/pdf/2409.17566
• Copy Paste: [[2409.17566]] Flexiffusion: Segment-wise Neural Architecture Search for Flexible Denoising Schedule(https://arxiv.org/abs/2409.17566)
• Keywords: diffusion, generative
• Abstract: Diffusion models are cutting-edge generative models adept at producing diverse, high-quality images. Despite their effectiveness, these models often require significant computational resources owing to their numerous sequential denoising steps and the significant inference cost of each step. Recently, Neural Architecture Search (NAS) techniques have been employed to automatically search for faster generation processes. However, NAS for diffusion is inherently time-consuming as it requires estimating thousands of diffusion models to search for the optimal one. In this paper, we introduce Flexiffusion, a novel training-free NAS paradigm designed to accelerate diffusion models by concurrently optimizing generation steps and network structures. Specifically, we partition the generation process into isometric step segments, each sequentially composed of a full step, multiple partial steps, and several null steps. The full step computes all network blocks, while the partial step involves part of the blocks, and the null step entails no computation. Flexiffusion autonomously explores flexible step combinations for each segment, substantially reducing search costs and enabling greater acceleration compared to the state-of-the-art (SOTA) method for diffusion models. Our searched models reported speedup factors of $2.6\times$ and $1.5\times$ for the original LDM-4-G and the SOTA, respectively. The factors for Stable Diffusion V1.5 and the SOTA are $5.1\times$ and $2.0\times$. We also verified the performance of Flexiffusion on multiple datasets, and positive experiment results indicate that Flexiffusion can effectively reduce redundancy in diffusion models.

Title: ID$^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition

• Authors: Shen Li, Jianqing Xu, Jiaying Wu, Miao Xiong, Ailin Deng, Jiazhen Ji, Yuge Huang, Wenjie Feng, Shouhong Ding, Bryan Hooi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17576
• Pdf URL: https://arxiv.org/pdf/2409.17576
• Copy Paste: [[2409.17576]] ID$^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition(https://arxiv.org/abs/2409.17576)
• Keywords: privacy, diffusion
• Abstract: Synthetic face recognition (SFR) aims to generate synthetic face datasets that mimic the distribution of real face data, which allows for training face recognition models in a privacy-preserving manner. Despite the remarkable potential of diffusion models in image generation, current diffusion-based SFR models struggle with generalization to real-world faces. To address this limitation, we outline three key objectives for SFR: (1) promoting diversity across identities (inter-class diversity), (2) ensuring diversity within each identity by injecting various facial attributes (intra-class diversity), and (3) maintaining identity consistency within each identity group (intra-class identity preservation). Inspired by these goals, we introduce a diffusion-fueled SFR model termed $\text{ID}^3$. $\text{ID}^3$ employs an ID-preserving loss to generate diverse yet identity-consistent facial appearances. Theoretically, we show that minimizing this loss is equivalent to maximizing the lower bound of an adjusted conditional log-likelihood over ID-preserving data. This equivalence motivates an ID-preserving sampling algorithm, which operates over an adjusted gradient vector field, enabling the generation of fake face recognition datasets that approximate the distribution of real-world faces. Extensive experiments across five challenging benchmarks validate the advantages of $\text{ID}^3$.

Title: Multimodal Banking Dataset: Understanding Client Needs through Event Sequences

• Authors: Mollaev Dzhambulat, Alexander Kostin, Postnova Maria, Ivan Karpukhin, Ivan A Kireev, Gleb Gusev, Andrey Savchenko
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17587
• Pdf URL: https://arxiv.org/pdf/2409.17587
• Copy Paste: [[2409.17587]] Multimodal Banking Dataset: Understanding Client Needs through Event Sequences(https://arxiv.org/abs/2409.17587)
• Keywords: privacy
• Abstract: Financial organizations collect a huge amount of data about clients that typically has a temporal (sequential) structure and is collected from various sources (modalities). Due to privacy issues, there are no large-scale open-source multimodal datasets of event sequences, which significantly limits the research in this area. In this paper, we present the industrial-scale publicly available multimodal banking dataset, MBD, that contains more than 1.5M corporate clients with several modalities: 950M bank transactions, 1B geo position events, 5M embeddings of dialogues with technical support and monthly aggregated purchases of four bank's products. All entries are properly anonymized from real proprietary bank data. Using this dataset, we introduce a novel benchmark with two business tasks: campaigning (purchase prediction in the next month) and matching of clients. We provide numerical results that demonstrate the superiority of our multi-modal baselines over single-modal techniques for each task. As a result, the proposed dataset can open new perspectives and facilitate the future development of practically important large-scale multimodal algorithms for event sequences. HuggingFace Link: this https URL Github Link: this https URL

Title: DualCoTs: Dual Chain-of-Thoughts Prompting for Sentiment Lexicon Expansion of Idioms

• Authors: Fuqiang Niu, Minghuan Tan, Bowen Zhang, Min Yang, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17588
• Pdf URL: https://arxiv.org/pdf/2409.17588
• Copy Paste: [[2409.17588]] DualCoTs: Dual Chain-of-Thoughts Prompting for Sentiment Lexicon Expansion of Idioms(https://arxiv.org/abs/2409.17588)
• Keywords: large language model
• Abstract: Idioms represent a ubiquitous vehicle for conveying sentiments in the realm of everyday discourse, rendering the nuanced analysis of idiom sentiment crucial for a comprehensive understanding of emotional expression within real-world texts. Nevertheless, the existing corpora dedicated to idiom sentiment analysis considerably limit research in text sentiment analysis. In this paper, we propose an innovative approach to automatically expand the sentiment lexicon for idioms, leveraging the capabilities of large language models through the application of Chain-of-Thought prompting. To demonstrate the effectiveness of this approach, we integrate multiple existing resources and construct an emotional idiom lexicon expansion dataset (called EmoIdiomE), which encompasses a comprehensive repository of Chinese and English idioms. Then we designed the Dual Chain-of-Thoughts (DualCoTs) method, which combines insights from linguistics and psycholinguistics, to demonstrate the effectiveness of using large models to automatically expand the sentiment lexicon for idioms. Experiments show that DualCoTs is effective in idioms sentiment lexicon expansion in both Chinese and English. For reproducibility, we will release the data and code upon acceptance.

Title: Improving Fast Adversarial Training via Self-Knowledge Guidance

• Authors: Chengze Jiang, Junkai Wang, Minjing Dong, Jie Gui, Xinli Shi, Yuan Cao, Yuan Yan Tang, James Tin-Yau Kwok
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17589
• Pdf URL: https://arxiv.org/pdf/2409.17589
• Copy Paste: [[2409.17589]] Improving Fast Adversarial Training via Self-Knowledge Guidance(https://arxiv.org/abs/2409.17589)
• Keywords: attack, robust
• Abstract: Adversarial training has achieved remarkable advancements in defending against adversarial attacks. Among them, fast adversarial training (FAT) is gaining attention for its ability to achieve competitive robustness with fewer computing resources. Existing FAT methods typically employ a uniform strategy that optimizes all training data equally without considering the influence of different examples, which leads to an imbalanced optimization. However, this imbalance remains unexplored in the field of FAT. In this paper, we conduct a comprehensive study of the imbalance issue in FAT and observe an obvious class disparity regarding their performances. This disparity could be embodied from a perspective of alignment between clean and robust accuracy. Based on the analysis, we mainly attribute the observed misalignment and disparity to the imbalanced optimization in FAT, which motivates us to optimize different training data adaptively to enhance robustness. Specifically, we take disparity and misalignment into consideration. First, we introduce self-knowledge guided regularization, which assigns differentiated regularization weights to each class based on its training state, alleviating class disparity. Additionally, we propose self-knowledge guided label relaxation, which adjusts label relaxation according to the training accuracy, alleviating the misalignment and improving robustness. By combining these methods, we formulate the Self-Knowledge Guided FAT (SKG-FAT), leveraging naturally generated knowledge during training to enhance the adversarial robustness without compromising training efficiency. Extensive experiments on four standard datasets demonstrate that the SKG-FAT improves the robustness and preserves competitive clean accuracy, outperforming the state-of-the-art methods.

Title: AsIf: Asset Interface Analysis of Industrial Automation Devices

• Authors: Thomas Rosenstatter, Christian Schäfer, Olaf Saßnick, Stefan Huber
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17593
• Pdf URL: https://arxiv.org/pdf/2409.17593
• Copy Paste: [[2409.17593]] AsIf: Asset Interface Analysis of Industrial Automation Devices(https://arxiv.org/abs/2409.17593)
• Keywords: security
• Abstract: As Industry 4.0 and the Industrial Internet of Things continue to advance, industrial control systems are increasingly adopting IT solutions, including communication standards and protocols. As these systems become more decentralized and interconnected, a critical need for enhanced security measures arises. Threat modeling is traditionally performed in structured brainstorming sessions involving domain and security experts. Such sessions, however, often fail to provide an exhaustive identification of assets and interfaces due to the lack of a systematic approach. This is a major issue, as it leads to poor threat modeling, resulting in insufficient mitigation strategies and, lastly, a flawed security architecture. We propose a method for the analysis of assets in industrial systems, with special focus on physical threats. Inspired by the ISO/OSI reference model, a systematic approach is introduced to help identify and classify asset interfaces. This results in an enriched system model of the asset, offering a comprehensive overview visually represented as an interface tree, thereby laying the foundation for subsequent threat modeling steps. To demonstrate the proposed method, the results of its application to a programmable logic controller (PLC) are presented. In support of this, a study involving a group of 12 security experts was conducted. Additionally, the study offers valuable insights into the experts' general perspectives and workflows on threat modeling.

Title: Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution

• Authors: Zhenyu Hu, Wanjie Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17597
• Pdf URL: https://arxiv.org/pdf/2409.17597
• Copy Paste: [[2409.17597]] Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution(https://arxiv.org/abs/2409.17597)
• Keywords: transformer
• Abstract: Window-based transformers have demonstrated outstanding performance in super-resolution tasks due to their adaptive modeling capabilities through local self-attention (SA). However, they exhibit higher computational complexity and inference latency than convolutional neural networks. In this paper, we first identify that the adaptability of the Transformers is derived from their adaptive spatial aggregation and advanced structural design, while their high latency results from the computational costs and memory layout transformations associated with the local SA. To simulate this aggregation approach, we propose an effective convolution-based linear focal separable attention (FSA), allowing for long-range dynamic modeling with linear complexity. Additionally, we introduce an effective dual-branch structure combined with an ultra-lightweight information exchange module (IEM) to enhance the aggregation of information by the Token Mixer. Finally, with respect to the structure, we modify the existing spatial-gate-based feedforward neural networks by incorporating a self-gate mechanism to preserve high-dimensional channel information, enabling the modeling of more complex relationships. With these advancements, we construct a convolution-based Transformer framework named the linear adaptive mixer network (LAMNet). Extensive experiments demonstrate that LAMNet achieves better performance than existing SA-based Transformer methods while maintaining the computational efficiency of convolutional neural networks, which can achieve a \(3\times\) speedup of inference time. The code will be publicly available at: this https URL.

Title: TA-Cleaner: A Fine-grained Text Alignment Backdoor Defense Strategy for Multimodal Contrastive Learning

• Authors: Yuan Xun, Siyuan Liang, Xiaojun Jia, Xinwei Liu, Xiaochun Cao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17601
• Pdf URL: https://arxiv.org/pdf/2409.17601
• Copy Paste: [[2409.17601]] TA-Cleaner: A Fine-grained Text Alignment Backdoor Defense Strategy for Multimodal Contrastive Learning(https://arxiv.org/abs/2409.17601)
• Keywords: defense, attack
• Abstract: Pre-trained large models for multimodal contrastive learning, such as CLIP, have been widely recognized in the industry as highly susceptible to data-poisoned backdoor attacks. This poses significant risks to downstream model training. In response to such potential threats, finetuning offers a simpler and more efficient defense choice compared to retraining large models with augmented data. In the supervised learning domain, fine-tuning defense strategies can achieve excellent defense performance. However, in the unsupervised and semi-supervised domain, we find that when CLIP faces some complex attack techniques, the existing fine-tuning defense strategy, CleanCLIP, has some limitations on defense performance. The synonym substitution of its text-augmentation is insufficient to enhance the text feature space. To compensate for this weakness, we improve it by proposing a fine-grained \textbf{T}ext \textbf{A}lignment \textbf{C}leaner (TA-Cleaner) to cut off feature connections of backdoor triggers. We randomly select a few samples for positive and negative subtext generation at each epoch of CleanCLIP, and align the subtexts to the images to strengthen the text self-supervision. We evaluate the effectiveness of our TA-Cleaner against six attack algorithms and conduct comprehensive zero-shot classification tests on ImageNet1K. Our experimental results demonstrate that TA-Cleaner achieves state-of-the-art defensiveness among finetuning-based defense techniques. Even when faced with the novel attack technique BadCLIP, our TA-Cleaner outperforms CleanCLIP by reducing the ASR of Top-1 and Top-10 by 52.02\% and 63.88\%, respectively.

Title: RmGPT: Rotating Machinery Generative Pretrained Model

• Authors: Yilin Wang, Yifei Yu, Kong Sun, Peixuan Lei, Yuxuan Zhang, Enrico Zio, Aiguo Xia, Yuanxiang Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17604
• Pdf URL: https://arxiv.org/pdf/2409.17604
• Copy Paste: [[2409.17604]] RmGPT: Rotating Machinery Generative Pretrained Model(https://arxiv.org/abs/2409.17604)
• Keywords: robust, extraction, generative
• Abstract: In industry, the reliability of rotating machinery is critical for production efficiency and safety. Current methods of Prognostics and Health Management (PHM) often rely on task-specific models, which face significant challenges in handling diverse datasets with varying signal characteristics, fault modes and operating conditions. Inspired by advancements in generative pretrained models, we propose RmGPT, a unified model for diagnosis and prognosis tasks. RmGPT introduces a novel token-based framework, incorporating Signal Tokens, Prompt Tokens, Time-Frequency Task Tokens and Fault Tokens to handle heterogeneous data within a unified model architecture. We leverage self-supervised learning for robust feature extraction and introduce a next signal token prediction pretraining strategy, alongside efficient prompt learning for task-specific adaptation. Extensive experiments demonstrate that RmGPT significantly outperforms state-of-the-art algorithms, achieving near-perfect accuracy in diagnosis tasks and exceptionally low errors in prognosis tasks. Notably, RmGPT excels in few-shot learning scenarios, achieving 92% accuracy in 16-class one-shot experiments, highlighting its adaptability and robustness. This work establishes RmGPT as a powerful PHM foundation model for rotating machinery, advancing the scalability and generalizability of PHM solutions.

Title: Good Data Is All Imitation Learning Needs

• Authors: Amir Samadi, Konstantinos Koufos, Kurt Debattista, Mehrdad Dianati
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17605
• Pdf URL: https://arxiv.org/pdf/2409.17605
• Copy Paste: [[2409.17605]] Good Data Is All Imitation Learning Needs(https://arxiv.org/abs/2409.17605)
• Keywords: robust
• Abstract: In this paper, we address the limitations of traditional teacher-student models, imitation learning, and behaviour cloning in the context of Autonomous/Automated Driving Systems (ADS), where these methods often struggle with incomplete coverage of real-world scenarios. To enhance the robustness of such models, we introduce the use of Counterfactual Explanations (CFEs) as a novel data augmentation technique for end-to-end ADS. CFEs, by generating training samples near decision boundaries through minimal input modifications, lead to a more comprehensive representation of expert driver strategies, particularly in safety-critical scenarios. This approach can therefore help improve the model's ability to handle rare and challenging driving events, such as anticipating darting out pedestrians, ultimately leading to safer and more trustworthy decision-making for ADS. Our experiments in the CARLA simulator demonstrate that CF-Driver outperforms the current state-of-the-art method, achieving a higher driving score and lower infraction rates. Specifically, CF-Driver attains a driving score of 84.2, surpassing the previous best model by 15.02 percentage points. These results highlight the effectiveness of incorporating CFEs in training end-to-end ADS. To foster further research, the CF-Driver code is made publicly available.

Title: ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue

• Authors: Zhangpu Li, Changhong Zou, Suxue Ma, Zhicheng Yang, Chen Du, Youbao Tang, Zhenjie Cao, Ning Zhang, Jui-Hsin Lai, Ruei-Sung Lin, Yuan Ni, Xingzhi Sun, Jing Xiao, Kai Zhang, Mei Han
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17610
• Pdf URL: https://arxiv.org/pdf/2409.17610
• Copy Paste: [[2409.17610]] ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue(https://arxiv.org/abs/2409.17610)
• Keywords: large language model
• Abstract: The rocketing prosperity of large language models (LLMs) in recent years has boosted the prevalence of vision-language models (VLMs) in the medical sector. In our online medical consultation scenario, a doctor responds to the texts and images provided by a patient in multiple rounds to diagnose her/his health condition, forming a multi-turn multimodal medical dialogue format. Unlike high-quality images captured by professional equipment in traditional medical visual question answering (Med-VQA), the images in our case are taken by patients' mobile phones. These images have poor quality control, with issues such as excessive background elements and the lesion area being significantly off-center, leading to degradation of vision-language alignment in the model training phase. In this paper, we propose ZALM3, a Zero-shot strategy to improve vision-language ALignment in Multi-turn Multimodal Medical dialogue. Since we observe that the preceding text conversations before an image can infer the regions of interest (RoIs) in the image, ZALM3 employs an LLM to summarize the keywords from the preceding context and a visual grounding model to extract the RoIs. The updated images eliminate unnecessary background noise and provide more effective vision-language alignment. To better evaluate our proposed method, we design a new subjective assessment metric for multi-turn unimodal/multimodal medical dialogue to provide a fine-grained performance comparison. Our experiments across three different clinical departments remarkably demonstrate the efficacy of ZALM3 with statistical significance.

Title: Benign or Not-Benign Overfitting in Token Selection of Attention Mechanism

• Authors: Keitaro Sakamoto, Issei Sato
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17625
• Pdf URL: https://arxiv.org/pdf/2409.17625
• Copy Paste: [[2409.17625]] Benign or Not-Benign Overfitting in Token Selection of Attention Mechanism(https://arxiv.org/abs/2409.17625)
• Keywords: transformer
• Abstract: Modern over-parameterized neural networks can be trained to fit the training data perfectly while still maintaining a high generalization performance. This "benign overfitting" phenomenon has been studied in a surge of recent theoretical work; however, most of these studies have been limited to linear models or two-layer neural networks. In this work, we analyze benign overfitting in the token selection mechanism of the attention architecture, which characterizes the success of transformer models. We first show the existence of a benign overfitting solution and explain its mechanism in the attention architecture. Next, we discuss whether the model converges to such a solution, raising the difficulties specific to the attention architecture. We then present benign overfitting cases and not-benign overfitting cases by conditioning different scenarios based on the behavior of attention probabilities during training. To the best of our knowledge, this is the first study to characterize benign overfitting for the attention mechanism.

Title: T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task

• Authors: Xindi Tong, Yujin Zhu, Shijian Fan, Liang Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17640
• Pdf URL: https://arxiv.org/pdf/2409.17640
• Copy Paste: [[2409.17640]] T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task(https://arxiv.org/abs/2409.17640)
• Keywords: fair, large language model
• Abstract: Long text summarization, gradually being essential for efficiently processing large volumes of information, stays challenging for Large Language Models (LLMs) such as GPT and LLaMA families because of the insufficient open-sourced training datasets and the high requirement of contextual details dealing. To address the issue, we design a novel zero-shot transfer learning framework, abbreviated as T3, to iteratively training a baseline LLM on an assistant task for the target task, where the former should own richer data resources and share structural or semantic similarity with the latter. In practice, T3 is approached to deal with the long text summarization task by utilizing question answering as the assistant task, and further validated its effectiveness on the BBC summary, NarraSum, FairytaleQA, and NLQuAD datasets, with up to nearly 14% improvement in ROUGE, 35% improvement in BLEU, and 16% improvement in Factscore compared to three baseline LLMs, demonstrating its potential for more assistant-target task combinations.

Title: Efficient In-Domain Question Answering for Resource-Constrained Environments

• Authors: Isaac Chung, Phat Vo, Arman Kizilkale, Aaron Reite
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17648
• Pdf URL: https://arxiv.org/pdf/2409.17648
• Copy Paste: [[2409.17648]] Efficient In-Domain Question Answering for Resource-Constrained Environments(https://arxiv.org/abs/2409.17648)
• Keywords: robust, large language model
• Abstract: Retrieval Augmented Generation (RAG) is a common method for integrating external knowledge into pretrained Large Language Models (LLMs) to enhance accuracy and relevancy in question answering (QA) tasks. However, prompt engineering and resource efficiency remain significant bottlenecks in developing optimal and robust RAG solutions for real-world QA applications. Recent studies have shown success in using fine tuning to address these problems; in particular, Retrieval Augmented Fine Tuning (RAFT) applied to smaller 7B models has demonstrated superior performance compared to RAG setups with much larger models such as GPT-3.5. The combination of RAFT with parameter-efficient fine tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA), promises an even more efficient solution, yet remains an unexplored area. In this work, we combine RAFT with LoRA to reduce fine tuning and storage requirements and gain faster inference times while maintaining comparable RAG performance. This results in a more compute-efficient RAFT, or CRAFT, which is particularly useful for knowledge-intensive QA tasks in resource-constrained environments where internet access may be restricted and hardware resources limited.

Title: Provable Performance Guarantees of Copy Detection Patterns

• Authors: Joakim Tutt, Slava Voloshynovskiy
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17649
• Pdf URL: https://arxiv.org/pdf/2409.17649
• Copy Paste: [[2409.17649]] Provable Performance Guarantees of Copy Detection Patterns(https://arxiv.org/abs/2409.17649)
• Keywords: security
• Abstract: Copy Detection Patterns (CDPs) are crucial elements in modern security applications, playing a vital role in safeguarding industries such as food, pharmaceuticals, and cosmetics. Current performance evaluations of CDPs predominantly rely on empirical setups using simplistic metrics like Hamming distances or Pearson correlation. These methods are often inadequate due to their sensitivity to distortions, degradation, and their limitations to stationary statistics of printing and imaging. Additionally, machine learning-based approaches suffer from distribution biases and fail to generalize to unseen counterfeit samples. Given the critical importance of CDPs in preventing counterfeiting, including the counterfeit vaccines issue highlighted during the COVID-19 pandemic, there is an urgent need for provable performance guarantees across various criteria. This paper aims to establish a theoretical framework to derive optimal criteria for the analysis, optimization, and future development of CDP authentication technologies, ensuring their reliability and effectiveness in diverse security scenarios.

Title: A Comprehensive Review of TLSNotary Protocol

• Authors: Maciej Kalka, Marek Kirejczyk
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17670
• Pdf URL: https://arxiv.org/pdf/2409.17670
• Copy Paste: [[2409.17670]] A Comprehensive Review of TLSNotary Protocol(https://arxiv.org/abs/2409.17670)
• Keywords: secure, security
• Abstract: Transport Layer Security (TLS) protocol is a cryptographic protocol designed to secure communication over the internet. The TLS protocol has become a fundamental in secure communication, most commonly used for securing web browsing sessions. In this work, we investigate the TLSNotary protocol, which aim to enable the Client to obtain proof of provenance for data from TLS session, while getting as much as possible from the TLS security properties. To achieve such proofs without any Server-side adjustments or permissions, the power of secure multi-party computation (MPC) together with zero knowledge proofs is used to extend the standard TLS Protocol. To make the compliacted landscape of MPC as comprehensible as possible we first introduce the cryptographic primitives required to understand the TLSNotary protocol and go through standard TLS protocol. Finally, we look at the TLSNotary protocol in detail.

Title: Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation

• Authors: Huan Yang, Jiahui Chen, Chaofan Ding, Runhua Shi, Siyu Xiong, Qingqi Hong, Xiaoqi Mo, Xinhan Di
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17674
• Pdf URL: https://arxiv.org/pdf/2409.17674
• Copy Paste: [[2409.17674]] Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation(https://arxiv.org/abs/2409.17674)
• Keywords: diffusion
• Abstract: Gestures are pivotal in enhancing co-speech communication. While recent works have mostly focused on point-level motion transformation or fully supervised motion representations through data-driven approaches, we explore the representation of gestures in co-speech, with a focus on self-supervised representation and pixel-level motion deviation, utilizing a diffusion model which incorporates latent motion features. Our approach leverages self-supervised deviation in latent representation to facilitate hand gestures generation, which are crucial for generating realistic gesture videos. Results of our first experiment demonstrate that our method enhances the quality of generated videos, with an improvement from 2.7 to 4.5% for FGD, DIV, and FVD, and 8.1% for PSNR, 2.5% for SSIM over the current state-of-the-art methods.

Title: EM-Net: Efficient Channel and Frequency Learning with Mamba for 3D Medical Image Segmentation

• Authors: Ao Chang, Jiajun Zeng, Ruobing Huang, Dong Ni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17675
• Pdf URL: https://arxiv.org/pdf/2409.17675
• Copy Paste: [[2409.17675]] EM-Net: Efficient Channel and Frequency Learning with Mamba for 3D Medical Image Segmentation(https://arxiv.org/abs/2409.17675)
• Keywords: transformer, segmentation
• Abstract: Convolutional neural networks have primarily led 3D medical image segmentation but may be limited by small receptive fields. Transformer models excel in capturing global relationships through self-attention but are challenged by high computational costs at high resolutions. Recently, Mamba, a state space model, has emerged as an effective approach for sequential modeling. Inspired by its success, we introduce a novel Mamba-based 3D medical image segmentation model called EM-Net. It not only efficiently captures attentive interaction between regions by integrating and selecting channels, but also effectively utilizes frequency domain to harmonize the learning of features across varying scales, while accelerating training speed. Comprehensive experiments on two challenging multi-organ datasets with other state-of-the-art (SOTA) algorithms show that our method exhibits better segmentation accuracy while requiring nearly half the parameter size of SOTA models and 2x faster training speed.

Title: Optimal Memorization Capacity of Transformers

• Authors: Tokio Kajitsuka, Issei Sato
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17677
• Pdf URL: https://arxiv.org/pdf/2409.17677
• Copy Paste: [[2409.17677]] Optimal Memorization Capacity of Transformers(https://arxiv.org/abs/2409.17677)
• Keywords: transformer
• Abstract: Recent research in the field of machine learning has increasingly focused on the memorization capacity of Transformers, but how efficient they are is not yet well understood. We demonstrate that Transformers can memorize labels with $\tilde{O}(\sqrt{N})$ parameters in a next-token prediction setting for $N$ input sequences of length $n$, which is proved to be optimal up to logarithmic factors. This indicates that Transformers can efficiently perform memorization with little influence from the input length $n$ owing to the benefit of parameter sharing. We also analyze the memorization capacity in the sequence-to-sequence setting, and find that $\tilde{O}(\sqrt{nN})$ parameters are not only sufficient, but also necessary at least for Transformers with hardmax. These results suggest that while self-attention mechanisms can efficiently identify input sequences, the feed-forward network becomes a bottleneck when associating a label to each token.

Title: Dark Miner: Defend against unsafe generation for text-to-image diffusion models

• Authors: Zheling Meng, Bo Peng, Xiaochuan Jin, Yue Jiang, Jing Dong, Wei Wang, Tieniu Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17682
• Pdf URL: https://arxiv.org/pdf/2409.17682
• Copy Paste: [[2409.17682]] Dark Miner: Defend against unsafe generation for text-to-image diffusion models(https://arxiv.org/abs/2409.17682)
• Keywords: defense, attack, diffusion
• Abstract: Text-to-image diffusion models have been demonstrated with unsafe generation due to unfiltered large-scale training data, such as violent, sexual, and shocking images, necessitating the erasure of unsafe concepts. Most existing methods focus on modifying the generation probabilities conditioned on the texts containing unsafe descriptions. However, they fail to guarantee safe generation for unseen texts in the training phase, especially for the prompts from adversarial attacks. In this paper, we re-analyze the erasure task and point out that existing methods cannot guarantee the minimization of the total probabilities of unsafe generation. To tackle this problem, we propose Dark Miner. It entails a recurring three-stage process that comprises mining, verifying, and circumventing. It greedily mines embeddings with maximum generation probabilities of unsafe concepts and reduces unsafe generation more effectively. In the experiments, we evaluate its performance on two inappropriate concepts, two objects, and two styles. Compared with 6 previous state-of-the-art methods, our method achieves better erasure and defense results in most cases, especially under 4 state-of-the-art attacks, while preserving the model's native generation capability. Our code will be available on GitHub.

Title: Zero- and Few-shot Named Entity Recognition and Text Expansion in Medication Prescriptions using ChatGPT

• Authors: Natthanaphop Isaradech, Andrea Riedel, Wachiranun Sirikul, Markus Kreuzthaler, Stefan Schulz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17683
• Pdf URL: https://arxiv.org/pdf/2409.17683
• Copy Paste: [[2409.17683]] Zero- and Few-shot Named Entity Recognition and Text Expansion in Medication Prescriptions using ChatGPT(https://arxiv.org/abs/2409.17683)
• Keywords: large language model
• Abstract: Introduction: Medication prescriptions are often in free text and include a mix of two languages, local brand names, and a wide range of idiosyncratic formats and abbreviations. Large language models (LLMs) have shown promising ability to generate text in response to input prompts. We use ChatGPT 3.5 to automatically structure and expand medication statements in discharge summaries and thus make them easier to interpret for people and machines. Methods: Named-entity Recognition (NER) and Text Expansion (EX) are used in a zero- and few-shot setting with different prompt strategies. 100 medication statements were manually annotated and curated. NER performance was measured by using strict and partial matching. For the task EX, two experts interpreted the results by assessing semantic equivalence between original and expanded statements. The model performance was measured by precision, recall, and F1 score. Results: For NER, the best-performing prompt reached an average F1 score of 0.94 in the test set. For EX, the few-shot prompt showed superior performance among other prompts, with an average F1 score of 0.87. Conclusion: Our study demonstrates good performance for NER and EX tasks in free-text medication statements using ChatGPT. Compared to a zero-shot baseline, a few-shot approach prevented the system from hallucinating, which would be unacceptable when processing safety-relevant medication data.

Title: Efficient Bias Mitigation Without Privileged Information

• Authors: Mateo Espinosa Zarlenga, Swami Sankaranarayanan, Jerone T. A. Andrews, Zohreh Shams, Mateja Jamnik, Alice Xiang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17691
• Pdf URL: https://arxiv.org/pdf/2409.17691
• Copy Paste: [[2409.17691]] Efficient Bias Mitigation Without Privileged Information(https://arxiv.org/abs/2409.17691)
• Keywords: robust
• Abstract: Deep neural networks trained via empirical risk minimisation often exhibit significant performance disparities across groups, particularly when group and task labels are spuriously correlated (e.g., "grassy background" and "cows"). Existing bias mitigation methods that aim to address this issue often either rely on group labels for training or validation, or require an extensive hyperparameter search. Such data and computational requirements hinder the practical deployment of these methods, especially when datasets are too large to be group-annotated, computational resources are limited, and models are trained through already complex pipelines. In this paper, we propose Targeted Augmentations for Bias Mitigation (TAB), a simple hyperparameter-free framework that leverages the entire training history of a helper model to identify spurious samples, and generate a group-balanced training set from which a robust model can be trained. We show that TAB improves worst-group performance without any group information or model selection, outperforming existing methods while maintaining overall accuracy.

Title: MIO: A Foundation Model on Multimodal Tokens

• Authors: Zekun Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jiashuo Wang, Ning Shi, Siyu Li, Yizhi Li, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, Wenhao Huang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17692
• Pdf URL: https://arxiv.org/pdf/2409.17692
• Copy Paste: [[2409.17692]] MIO: A Foundation Model on Multimodal Tokens(https://arxiv.org/abs/2409.17692)
• Keywords: large language model
• Abstract: In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. While the emergence of large language models (LLMs) and multimodal large language models (MM-LLMs) propels advancements in artificial general intelligence through their versatile capabilities, they still lack true any-to-any understanding and generation. Recently, the release of GPT-4o has showcased the remarkable potential of any-to-any LLMs for complex real-world tasks, enabling omnidirectional input and output across images, speech, and text. However, it is closed-source and does not support the generation of multimodal interleaved sequences. To address this gap, we present MIO, which is trained on a mixture of discrete tokens across four modalities using causal multimodal modeling. MIO undergoes a four-stage training process: (1) alignment pre-training, (2) interleaved pre-training, (3) speech-enhanced pre-training, and (4) comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. Our experimental results indicate that MIO exhibits competitive, and in some cases superior, performance compared to previous dual-modal baselines, any-to-any model baselines, and even modality-specific baselines. Moreover, MIO demonstrates advanced capabilities inherent to its any-to-any feature, such as interleaved video-text generation, chain-of-visual-thought reasoning, visual guideline generation, instructional image editing, etc.

Title: MoJE: Mixture of Jailbreak Experts, Naive Tabular Classifiers as Guard for Prompt Attacks

• Authors: Giandomenico Cornacchia, Giulio Zizzo, Kieran Fraser, Muhammad Zaid Hamed, Ambrish Rawat, Mark Purcell
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17699
• Pdf URL: https://arxiv.org/pdf/2409.17699
• Copy Paste: [[2409.17699]] MoJE: Mixture of Jailbreak Experts, Naive Tabular Classifiers as Guard for Prompt Attacks(https://arxiv.org/abs/2409.17699)
• Keywords: security, privacy, protect, attack, robust, large language model
• Abstract: The proliferation of Large Language Models (LLMs) in diverse applications underscores the pressing need for robust security measures to thwart potential jailbreak attacks. These attacks exploit vulnerabilities within LLMs, endanger data integrity and user privacy. Guardrails serve as crucial protective mechanisms against such threats, but existing models often fall short in terms of both detection accuracy, and computational efficiency. This paper advocates for the significance of jailbreak attack prevention on LLMs, and emphasises the role of input guardrails in safeguarding these models. We introduce MoJE (Mixture of Jailbreak Expert), a novel guardrail architecture designed to surpass current limitations in existing state-of-the-art guardrails. By employing simple linguistic statistical techniques, MoJE excels in detecting jailbreak attacks while maintaining minimal computational overhead during model inference. Through rigorous experimentation, MoJE demonstrates superior performance capable of detecting 90% of the attacks without compromising benign prompts, enhancing LLMs security against jailbreak attacks.

Title: PGN: The RNN's New Successor is Effective for Long-Range Time Series Forecasting

• Authors: Yuxin Jia, Youfang Lin, Jing Yu, Shuo Wang, Tianhao Liu, Huaiyu Wan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17703
• Pdf URL: https://arxiv.org/pdf/2409.17703
• Copy Paste: [[2409.17703]] PGN: The RNN's New Successor is Effective for Long-Range Time Series Forecasting(https://arxiv.org/abs/2409.17703)
• Keywords: extraction
• Abstract: Due to the recurrent structure of RNN, the long information propagation path poses limitations in capturing long-term dependencies, gradient explosion/vanishing issues, and inefficient sequential execution. Based on this, we propose a novel paradigm called Parallel Gated Network (PGN) as the new successor to RNN. PGN directly captures information from previous time steps through the designed Historical Information Extraction (HIE) layer and leverages gated mechanisms to select and fuse it with the current time step information. This reduces the information propagation path to $\mathcal{O}(1)$, effectively addressing the limitations of RNN. To enhance PGN's performance in long-range time series forecasting tasks, we propose a novel temporal modeling framework called Temporal PGN (TPGN). TPGN incorporates two branches to comprehensively capture the semantic information of time series. One branch utilizes PGN to capture long-term periodic patterns while preserving their local characteristics. The other branch employs patches to capture short-term information and aggregate the global representation of the series. TPGN achieves a theoretical complexity of $\mathcal{O}(\sqrt{L})$, ensuring efficiency in its operations. Experimental results on five benchmark datasets demonstrate the state-of-the-art (SOTA) performance and high efficiency of TPGN, further confirming the effectiveness of PGN as the new successor to RNN in long-range time series forecasting. The code is available in this repository: \url{this https URL}.

Title: Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit

• Authors: Dimitrios Kollias, Chunchang Shao, Odysseus Kaloidas, Ioannis Patras
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17717
• Pdf URL: https://arxiv.org/pdf/2409.17717
• Copy Paste: [[2409.17717]] Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit(https://arxiv.org/abs/2409.17717)
• Keywords: fair
• Abstract: In this paper, we introduce Behavior4All, a comprehensive, open-source toolkit for in-the-wild facial behavior analysis, integrating Face Localization, Valence-Arousal Estimation, Basic Expression Recognition and Action Unit Detection, all within a single framework. Available in both CPU-only and GPU-accelerated versions, Behavior4All leverages 12 large-scale, in-the-wild datasets consisting of over 5 million images from diverse demographic groups. It introduces a novel framework that leverages distribution matching and label co-annotation to address tasks with non-overlapping annotations, encoding prior knowledge of their relatedness. In the largest study of its kind, Behavior4All outperforms both state-of-the-art and toolkits in overall performance as well as fairness across all databases and tasks. It also demonstrates superior generalizability on unseen databases and on compound expression recognition. Finally, Behavior4All is way times faster than other toolkits.

Title: Recent advances in interpretable machine learning using structure-based protein representations

• Authors: Luiz Felipe Vecchietti, Minji Lee, Begench Hangeldiyev, Hyunkyu Jung, Hahnbeom Park, Tae-Kyun Kim, Meeyoung Cha, Ho Min Kim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17726
• Pdf URL: https://arxiv.org/pdf/2409.17726
• Copy Paste: [[2409.17726]] Recent advances in interpretable machine learning using structure-based protein representations(https://arxiv.org/abs/2409.17726)
• Keywords: interpretability
• Abstract: Recent advancements in machine learning (ML) are transforming the field of structural biology. For example, AlphaFold, a groundbreaking neural network for protein structure prediction, has been widely adopted by researchers. The availability of easy-to-use interfaces and interpretable outcomes from the neural network architecture, such as the confidence scores used to color the predicted structures, have made AlphaFold accessible even to non-ML experts. In this paper, we present various methods for representing protein 3D structures from low- to high-resolution, and show how interpretable ML methods can support tasks such as predicting protein structures, protein function, and protein-protein interactions. This survey also emphasizes the significance of interpreting and visualizing ML-based inference for structure-based protein representations that enhance interpretability and knowledge discovery. Developing such interpretable approaches promises to further accelerate fields including drug development and protein design.

Title: Neural Implicit Representation for Highly Dynamic LiDAR Mapping and Odometry

• Authors: Qi Zhang, He Wang, Ru Li, Wenbin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17729
• Pdf URL: https://arxiv.org/pdf/2409.17729
• Copy Paste: [[2409.17729]] Neural Implicit Representation for Highly Dynamic LiDAR Mapping and Odometry(https://arxiv.org/abs/2409.17729)
• Keywords: robust, segmentation
• Abstract: Recent advancements in Simultaneous Localization and Mapping (SLAM) have increasingly highlighted the robustness of LiDAR-based techniques. At the same time, Neural Radiance Fields (NeRF) have introduced new possibilities for 3D scene reconstruction, exemplified by SLAM systems. Among these, NeRF-LOAM has shown notable performance in NeRF-based SLAM applications. However, despite its strengths, these systems often encounter difficulties in dynamic outdoor environments due to their inherent static assumptions. To address these limitations, this paper proposes a novel method designed to improve reconstruction in highly dynamic outdoor scenes. Based on NeRF-LOAM, the proposed approach consists of two primary components. First, we separate the scene into static background and dynamic foreground. By identifying and excluding dynamic elements from the mapping process, this segmentation enables the creation of a dense 3D map that accurately represents the static background only. The second component extends the octree structure to support multi-resolution representation. This extension not only enhances reconstruction quality but also aids in the removal of dynamic objects identified by the first module. Additionally, Fourier feature encoding is applied to the sampled points, capturing high-frequency information and leading to more complete reconstruction results. Evaluations on various datasets demonstrate that our method achieves more competitive results compared to current state-of-the-art approaches.

Title: AnyLogo: Symbiotic Subject-Driven Diffusion System with Gemini Status

• Authors: Jinghao Zhang, Wen Qian, Hao Luo, Fan Wang, Feng Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17740
• Pdf URL: https://arxiv.org/pdf/2409.17740
• Copy Paste: [[2409.17740]] AnyLogo: Symbiotic Subject-Driven Diffusion System with Gemini Status(https://arxiv.org/abs/2409.17740)
• Keywords: extraction, diffusion
• Abstract: Diffusion models have made compelling progress on facilitating high-throughput daily production. Nevertheless, the appealing customized requirements are remain suffered from instance-level finetuning for authentic fidelity. Prior zero-shot customization works achieve the semantic consistence through the condensed injection of identity features, while addressing detailed low-level signatures through complex model configurations and subject-specific fabrications, which significantly break the statistical coherence within the overall system and limit the applicability across various scenarios. To facilitate the generic signature concentration with rectified efficiency, we present \textbf{AnyLogo}, a zero-shot region customizer with remarkable detail consistency, building upon the symbiotic diffusion system with eliminated cumbersome designs. Streamlined as vanilla image generation, we discern that the rigorous signature extraction and creative content generation are promisingly compatible and can be systematically recycled within a single denoising model. In place of the external configurations, the gemini status of the denoising model promote the reinforced subject transmission efficiency and disentangled semantic-signature space with continuous signature decoration. Moreover, the sparse recycling paradigm is adopted to prevent the duplicated risk with compressed transmission quota for diversified signature stimulation. Extensive experiments on constructed logo-level benchmarks demonstrate the effectiveness and practicability of our methods.

Title: Privacy for Quantum Annealing. Attack on Spin Reversal Transformations in the case of cryptanalysis

• Authors: Mateusz Leśniak, Michał Wroński
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17744
• Pdf URL: https://arxiv.org/pdf/2409.17744
• Copy Paste: [[2409.17744]] Privacy for Quantum Annealing. Attack on Spin Reversal Transformations in the case of cryptanalysis(https://arxiv.org/abs/2409.17744)
• Keywords: privacy, attack
• Abstract: This paper demonstrates that applying spin reversal transformations (SRT), commonly known as a sufficient method for privacy enhancing in problems solved using quantum annealing, does not guarantee privacy for all possible problems. We show how to recover the original problem from the Ising problem obtained using SRT when the resulting problem in Ising form represents the algebraic attack on the $E_0$ stream cipher. A small example is used to illustrate how to retrieve the original problem from the one transformed by SRT. Moreover, it is shown that our method is efficient even for full-scale problems.

Title: Text Image Generation for Low-Resource Languages with Dual Translation Learning

• Authors: Chihiro Noguchi, Shun Fukuda, Shoichiro Mihara, Masao Yamanaka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17747
• Pdf URL: https://arxiv.org/pdf/2409.17747
• Copy Paste: [[2409.17747]] Text Image Generation for Low-Resource Languages with Dual Translation Learning(https://arxiv.org/abs/2409.17747)
• Keywords: diffusion
• Abstract: Scene text recognition in low-resource languages frequently faces challenges due to the limited availability of training datasets derived from real-world scenes. This study proposes a novel approach that generates text images in low-resource languages by emulating the style of real text images from high-resource languages. Our approach utilizes a diffusion model that is conditioned on binary states: ``synthetic'' and ``real.'' The training of this model involves dual translation tasks, where it transforms plain text images into either synthetic or real text images, based on the binary states. This approach not only effectively differentiates between the two domains but also facilitates the model's explicit recognition of characters in the target language. Furthermore, to enhance the accuracy and variety of generated text images, we introduce two guidance techniques: Fidelity-Diversity Balancing Guidance and Fidelity Enhancement Guidance. Our experimental results demonstrate that the text images generated by our proposed framework can significantly improve the performance of scene text recognition models for low-resource languages.

Title: Byzantine-Robust Aggregation for Securing Decentralized Federated Learning

• Authors: Diego Cajaraville-Aboy, Ana Fernández-Vilas, Rebeca P. Díaz-Redondo, Manuel Fernández-Veiga
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17754
• Pdf URL: https://arxiv.org/pdf/2409.17754
• Copy Paste: [[2409.17754]] Byzantine-Robust Aggregation for Securing Decentralized Federated Learning(https://arxiv.org/abs/2409.17754)
• Keywords: security, privacy, attack, robust, federate
• Abstract: Federated Learning (FL) emerges as a distributed machine learning approach that addresses privacy concerns by training AI models locally on devices. Decentralized Federated Learning (DFL) extends the FL paradigm by eliminating the central server, thereby enhancing scalability and robustness through the avoidance of a single point of failure. However, DFL faces significant challenges in optimizing security, as most Byzantine-robust algorithms proposed in the literature are designed for centralized scenarios. In this paper, we present a novel Byzantine-robust aggregation algorithm to enhance the security of Decentralized Federated Learning environments, coined WFAgg. This proposal handles the adverse conditions and strength robustness of dynamic decentralized topologies at the same time by employing multiple filters to identify and mitigate Byzantine attacks. Experimental results demonstrate the effectiveness of the proposed algorithm in maintaining model accuracy and convergence in the presence of various Byzantine attack scenarios, outperforming state-of-the-art centralized Byzantine-robust aggregation schemes (such as Multi-Krum or Clustering). These algorithms are evaluated on an IID image classification problem in both centralized and decentralized scenarios.

Title: Confidence intervals uncovered: Are we ready for real-world medical imaging AI?

• Authors: Evangelia Christodoulou, Annika Reinke, Rola Houhou, Piotr Kalinowski, Selen Erkan, Carole H. Sudre, Ninon Burgos, Sofiène Boutaj, Sophie Loizillon, Maëlys Solal, Nicola Rieke, Veronika Cheplygina, Michela Antonelli, Leon D. Mayer, Minu D. Tizabi, M. Jorge Cardoso, Amber Simpson, Paul F. Jäger, Annette Kopp-Schneider, Gaël Varoquaux, Olivier Colliot, Lena Maier-Hein
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17763
• Pdf URL: https://arxiv.org/pdf/2409.17763
• Copy Paste: [[2409.17763]] Confidence intervals uncovered: Are we ready for real-world medical imaging AI?(https://arxiv.org/abs/2409.17763)
• Keywords: segmentation
• Abstract: Medical imaging is spearheading the AI transformation of healthcare. Performance reporting is key to determine which methods should be translated into clinical practice. Frequently, broad conclusions are simply derived from mean performance values. In this paper, we argue that this common practice is often a misleading simplification as it ignores performance variability. Our contribution is threefold. (1) Analyzing all MICCAI segmentation papers (n = 221) published in 2023, we first observe that more than 50\% of papers do not assess performance variability at all. Moreover, only one (0.5\%) paper reported confidence intervals (CIs) for model performance. (2) To address the reporting bottleneck, we show that the unreported standard deviation (SD) in segmentation papers can be approximated by a second-order polynomial function of the mean Dice similarity coefficient (DSC). Based on external validation data from 56 previous MICCAI challenges, we demonstrate that this approximation can accurately reconstruct the CI of a method using information provided in publications. (3) Finally, we reconstructed 95\% CIs around the mean DSC of MICCAI 2023 segmentation papers. The median CI width was 0.03 which is three times larger than the median performance gap between the first and second ranked method. For more than 60\% of papers, the mean performance of the second-ranked method was within the CI of the first-ranked method. We conclude that current publications typically do not provide sufficient evidence to support which models could potentially be translated into clinical practice.

Title: Federated Learning under Attack: Improving Gradient Inversion for Batch of Images

• Authors: Luiz Leite, Yuri Santo, Bruno L. Dalmazo, André Riker
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17767
• Pdf URL: https://arxiv.org/pdf/2409.17767
• Copy Paste: [[2409.17767]] Federated Learning under Attack: Improving Gradient Inversion for Batch of Images(https://arxiv.org/abs/2409.17767)
• Keywords: privacy, attack, federate
• Abstract: Federated Learning (FL) has emerged as a machine learning approach able to preserve the privacy of user's data. Applying FL, clients train machine learning models on a local dataset and a central server aggregates the learned parameters coming from the clients, training a global machine learning model without sharing user's data. However, the state-of-the-art shows several approaches to promote attacks on FL systems. For instance, inverting or leaking gradient attacks can find, with high precision, the local dataset used during the training phase of the FL. This paper presents an approach, called Deep Leakage from Gradients with Feedback Blending (DLG-FB), which is able to improve the inverting gradient attack, considering the spatial correlation that typically exists in batches of images. The performed evaluation shows an improvement of 19.18% and 48,82% in terms of attack success rate and the number of iterations per attacked image, respectively.

Title: Faithfulness and the Notion of Adversarial Sensitivity in NLP Explanations

• Authors: Supriya Manna, Niladri Sett
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17774
• Pdf URL: https://arxiv.org/pdf/2409.17774
• Copy Paste: [[2409.17774]] Faithfulness and the Notion of Adversarial Sensitivity in NLP Explanations(https://arxiv.org/abs/2409.17774)
• Keywords: attack
• Abstract: Faithfulness is arguably the most critical metric to assess the reliability of explainable AI. In NLP, current methods for faithfulness evaluation are fraught with discrepancies and biases, often failing to capture the true reasoning of models. We introduce Adversarial Sensitivity as a novel approach to faithfulness evaluation, focusing on the explainer's response when the model is under adversarial attack. Our method accounts for the faithfulness of explainers by capturing sensitivity to adversarial input changes. This work addresses significant limitations in existing evaluation techniques, and furthermore, quantifies faithfulness from a crucial yet underexplored paradigm.

Title: UNICORN: A Deep Learning Model for Integrating Multi-Stain Data in Histopathology

• Authors: Valentin Koch, Sabine Bauer, Valerio Luppberger, Michael Joner, Heribert Schunkert, Julia A. Schnabel, Moritz von Scheidt, Carsten Marr
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17775
• Pdf URL: https://arxiv.org/pdf/2409.17775
• Copy Paste: [[2409.17775]] UNICORN: A Deep Learning Model for Integrating Multi-Stain Data in Histopathology(https://arxiv.org/abs/2409.17775)
• Keywords: explainability, transformer
• Abstract: Background: The integration of multi-stain histopathology images through deep learning poses a significant challenge in digital histopathology. Current multi-modal approaches struggle with data heterogeneity and missing data. This study aims to overcome these limitations by developing a novel transformer model for multi-stain integration that can handle missing data during training as well as inference. Methods: We propose UNICORN (UNiversal modality Integration Network for CORonary classificatioN) a multi-modal transformer capable of processing multi-stain histopathology for atherosclerosis severity class prediction. The architecture comprises a two-stage, end-to-end trainable model with specialized modules utilizing transformer self-attention blocks. The initial stage employs domain-specific expert modules to extract features from each modality. In the subsequent stage, an aggregation expert module integrates these features by learning the interactions between the different data modalities. Results: Evaluation was performed using a multi-class dataset of atherosclerotic lesions from the Munich Cardiovascular Studies Biobank (MISSION), using over 4,000 paired multi-stain whole slide images (WSIs) from 170 deceased individuals on 7 prespecified segments of the coronary tree, each stained according to four histopathological protocols. UNICORN achieved a classification accuracy of 0.67, outperforming other state-of-the-art models. The model effectively identifies relevant tissue phenotypes across stainings and implicitly models disease progression. Conclusion: Our proposed multi-modal transformer model addresses key challenges in medical data analysis, including data heterogeneity and missing modalities. Explainability and the model's effectiveness in predicting atherosclerosis progression underscores its potential for broader applications in medical research.

Title: Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification

• Authors: Raja Kumar, Raghav Singhal, Pranamya Kulkarni, Deval Mehta, Kshitij Jadhav
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17777
• Pdf URL: https://arxiv.org/pdf/2409.17777
• Copy Paste: [[2409.17777]] Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification(https://arxiv.org/abs/2409.17777)
• Keywords: robust
• Abstract: Deep multimodal learning has shown remarkable success by leveraging contrastive learning to capture explicit one-to-one relations across modalities. However, real-world data often exhibits shared relations beyond simple pairwise associations. We propose M3CoL, a Multimodal Mixup Contrastive Learning approach to capture nuanced shared relations inherent in multimodal data. Our key contribution is a Mixup-based contrastive loss that learns robust representations by aligning mixed samples from one modality with their corresponding samples from other modalities thereby capturing shared relations between them. For multimodal classification tasks, we introduce a framework that integrates a fusion module with unimodal prediction modules for auxiliary supervision during training, complemented by our proposed Mixup-based contrastive loss. Through extensive experiments on diverse datasets (N24News, ROSMAP, BRCA, and Food-101), we demonstrate that M3CoL effectively captures shared multimodal relations and generalizes across domains. It outperforms state-of-the-art methods on N24News, ROSMAP, and BRCA, while achieving comparable performance on Food-101. Our work highlights the significance of learning shared relations for robust multimodal learning, opening up promising avenues for future research.

Title: Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs

• Authors: Qinpeng Cui, Yixuan Liu, Xinyi Zhang, Qiqi Bao, Zhongdao Wang, Qingmin Liao, Li Wang, Tian Lu, Emad Barsoum
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17778
• Pdf URL: https://arxiv.org/pdf/2409.17778
• Copy Paste: [[2409.17778]] Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs(https://arxiv.org/abs/2409.17778)
• Keywords: diffusion, generative
• Abstract: Diffusion-based image super-resolution (SR) models have attracted substantial interest due to their powerful image restoration capabilities. However, prevailing diffusion models often struggle to strike an optimal balance between efficiency and performance. Typically, they either neglect to exploit the potential of existing extensive pretrained models, limiting their generative capacity, or they necessitate a dozens of forward passes starting from random noises, compromising inference efficiency. In this paper, we present DoSSR, a Domain Shift diffusion-based SR model that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. At the core of our approach is a domain shift equation that integrates seamlessly with existing diffusion models. This integration not only improves the use of diffusion prior but also boosts inference efficiency. Moreover, we advance our method by transitioning the discrete shift process to a continuous formulation, termed as DoS-SDEs. This advancement leads to the fast and customized solvers that further enhance sampling efficiency. Empirical results demonstrate that our proposed method achieves state-of-the-art performance on synthetic and real-world datasets, while notably requiring only 5 sampling steps. Compared to previous diffusion prior based methods, our approach achieves a remarkable speedup of 5-7 times, demonstrating its superior efficiency. Code: this https URL.

Title: CASPFormer: Trajectory Prediction from BEV Images with Deformable Attention

• Authors: Harsh Yadav, Maximilian Schaefer, Kun Zhao, Tobias Meisen
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17790
• Pdf URL: https://arxiv.org/pdf/2409.17790
• Copy Paste: [[2409.17790]] CASPFormer: Trajectory Prediction from BEV Images with Deformable Attention(https://arxiv.org/abs/2409.17790)
• Keywords: transformer
• Abstract: Motion prediction is an important aspect for Autonomous Driving (AD) and Advance Driver Assistance Systems (ADAS). Current state-of-the-art motion prediction methods rely on High Definition (HD) maps for capturing the surrounding context of the ego vehicle. Such systems lack scalability in real-world deployment as HD maps are expensive to produce and update in real-time. To overcome this issue, we propose Context Aware Scene Prediction Transformer (CASPFormer), which can perform multi-modal motion prediction from rasterized Bird-Eye-View (BEV) images. Our system can be integrated with any upstream perception module that is capable of generating BEV images. Moreover, CASPFormer directly decodes vectorized trajectories without any postprocessing. Trajectories are decoded recurrently using deformable attention, as it is computationally efficient and provides the network with the ability to focus its attention on the important spatial locations of the BEV images. In addition, we also address the issue of mode collapse for generating multiple scene-consistent trajectories by incorporating learnable mode queries. We evaluate our model on the nuScenes dataset and show that it reaches state-of-the-art across multiple metrics

Title: Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness

• Authors: Jian Li, Haojing Huang, Yujia Zhang, Pengfei Xu, Xi Chen, Rui Song, Lida Shi, Jingwen Wang, Hao Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17791
• Pdf URL: https://arxiv.org/pdf/2409.17791
• Copy Paste: [[2409.17791]] Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness(https://arxiv.org/abs/2409.17791)
• Keywords: large language model
• Abstract: Recently, there has been significant interest in replacing the reward model in Reinforcement Learning with Human Feedback (RLHF) methods for Large Language Models (LLMs), such as Direct Preference Optimization (DPO) and its variants. These approaches commonly use a binary cross-entropy mechanism on pairwise samples, i.e., minimizing and maximizing the loss based on preferred or dis-preferred responses, respectively. However, while this training strategy omits the reward model, it also overlooks the varying preference degrees within different responses. We hypothesize that this is a key factor hindering LLMs from sufficiently understanding human preferences. To address this problem, we propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss, thereby helping LLMs improve their ability to understand the degree of preference. Extensive experiments are conducted on two widely used datasets of different tasks. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods and significantly boost their performance to achieve state-of-the-art performance. We also conduct detailed analyses to offer comprehensive insights into SPO, which verifies its effectiveness. The code is available at this https URL.

Title: Continual learning with task specialist

• Authors: Indu Solomon, Aye Phyu Phyu Aung, Uttam Kumar, Senthilnath Jayavelu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17806
• Pdf URL: https://arxiv.org/pdf/2409.17806
• Copy Paste: [[2409.17806]] Continual learning with task specialist(https://arxiv.org/abs/2409.17806)
• Keywords: diffusion
• Abstract: Continual learning (CL) adapt the deep learning scenarios with timely updated datasets. However, existing CL models suffer from the catastrophic forgetting issue, where new knowledge replaces past learning. In this paper, we propose Continual Learning with Task Specialists (CLTS) to address the issues of catastrophic forgetting and limited labelled data in real-world datasets by performing class incremental learning of the incoming stream of data. The model consists of Task Specialists (T S) and Task Predictor (T P ) with pre-trained Stable Diffusion (SD) module. Here, we introduce a new specialist to handle a new task sequence and each T S has three blocks; i) a variational autoencoder (V AE) to learn the task distribution in a low dimensional latent space, ii) a K-Means block to perform data clustering and iii) Bootstrapping Language-Image Pre-training (BLIP ) model to generate a small batch of captions from the input data. These captions are fed as input to the pre-trained stable diffusion model (SD) for the generation of task samples. The proposed model does not store any task samples for replay, instead uses generated samples from SD to train the T P module. A comparison study with four SOTA models conducted on three real-world datasets shows that the proposed model outperforms all the selected baselines

Title: Inference-Time Language Model Alignment via Integrated Value Guidance

• Authors: Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, Yu Qiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17819
• Pdf URL: https://arxiv.org/pdf/2409.17819
• Copy Paste: [[2409.17819]] Inference-Time Language Model Alignment via Integrated Value Guidance(https://arxiv.org/abs/2409.17819)
• Keywords: large language model
• Abstract: Large language models are typically fine-tuned to align with human preferences, but tuning large models is computationally intensive and complex. In this work, we introduce $\textit{Integrated Value Guidance}$ (IVG), a method that uses implicit and explicit value functions to guide language model decoding at token and chunk-level respectively, efficiently aligning large language models purely at inference time. This approach circumvents the complexities of direct fine-tuning and outperforms traditional methods. Empirically, we demonstrate the versatility of IVG across various tasks. In controlled sentiment generation and summarization tasks, our method significantly improves the alignment of large models using inference-time guidance from $\texttt{gpt2}$-based value functions. Moreover, in a more challenging instruction-following benchmark AlpacaEval 2.0, we show that both specifically tuned and off-the-shelf value functions greatly improve the length-controlled win rates of large models against $\texttt{gpt-4-turbo}$ (e.g., $19.51\% \rightarrow 26.51\%$ for $\texttt{Mistral-7B-Instruct-v0.2}$ and $25.58\% \rightarrow 33.75\%$ for $\texttt{Mixtral-8x7B-Instruct-v0.1}$ with Tulu guidance).

Title: Ordinary Differential Equations for Enhanced 12-Lead ECG Generation

• Authors: Yakir Yehuda, Kira Radinsky
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17833
• Pdf URL: https://arxiv.org/pdf/2409.17833
• Copy Paste: [[2409.17833]] Ordinary Differential Equations for Enhanced 12-Lead ECG Generation(https://arxiv.org/abs/2409.17833)
• Keywords: generative
• Abstract: In the realm of artificial intelligence, the generation of realistic training data for supervised learning tasks presents a significant challenge. This is particularly true in the synthesis of electrocardiograms (ECGs), where the objective is to develop a synthetic 12-lead ECG model. The primary complexity of this task stems from accurately modeling the intricate biological and physiological interactions among different ECG leads. Although mathematical process simulators have shed light on these dynamics, effectively incorporating this understanding into generative models is not straightforward. In this work, we introduce an innovative method that employs ordinary differential equations (ODEs) to enhance the fidelity of generating 12-lead ECG data. This approach integrates a system of ODEs that represent cardiac dynamics directly into the generative model's optimization process, allowing for the production of biologically plausible ECG training data that authentically reflects real-world variability and inter-lead dependencies. We conducted an empirical analysis of thousands of ECGs and found that incorporating cardiac simulation insights into the data generation process significantly improves the accuracy of heart abnormality classifiers trained on this synthetic 12-lead ECG data.

Title: PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdification

• Authors: Tianfang Xie, Tianjing Li, Wei Zhu, Wei Han, Yi Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17834
• Pdf URL: https://arxiv.org/pdf/2409.17834
• Copy Paste: [[2409.17834]] PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdification(https://arxiv.org/abs/2409.17834)
• Keywords: transformer, large language model
• Abstract: Due to their substantial sizes, large language models (LLMs) are typically deployed within a single-backbone multi-tenant framework. In this setup, a single instance of an LLM backbone must cater to multiple users or tasks through the application of various parameter-efficient fine-tuning (PEFT) models. Despite the availability of numerous effective PEFT techniques such as LoRA, there remains a need for a PEFT approach that achieves both high efficiency during inference and competitive performance on downstream tasks. In this research, we introduce a new and straightforward PEFT methodology named \underline{P}rompt D\underline{E}pen\underline{D}ent \underline{R}epresentation M\underline{O}dification (PEDRO). The proposed method involves integrating a lightweight vector generator into each Transformer layer, which generates vectors contingent upon the input prompts. These vectors then modify the hidden representations created by the LLM through a dot product operation, thereby influencing the semantic output and generated content of the model. Extensive experimentation across a variety of tasks indicates that: (a) PEDRO surpasses recent PEFT benchmarks when using a similar number of tunable parameters. (b) Under the single-backbone multi-tenant deployment model, PEDRO exhibits superior efficiency compared to LoRA, indicating significant industrial potential.

Title: Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models

• Authors: Hui-Po Wang, Mario Fritz
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17836
• Pdf URL: https://arxiv.org/pdf/2409.17836
• Copy Paste: [[2409.17836]] Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models(https://arxiv.org/abs/2409.17836)
• Keywords: large language model
• Abstract: Despite the widespread use of statistical prior models in various fields, such models for neural network gradients have long been overlooked. The inherent challenge stems from their high-dimensional structures and complex interdependencies, which complicate effective modeling. In this work, we demonstrate the potential of large language models (LLMs) to act as gradient priors in a zero-shot setting. We examine the property by considering lossless gradient compression -- a critical application in distributed learning -- that depends heavily on precise probability modeling. To achieve this, we introduce LM-GC, a novel method that integrates LLMs with arithmetic coding. Our technique converts plain gradients into text-like formats, enhancing token efficiency by up to 38 times compared to their plain representations. We ensure that this data conversion maintains a close alignment with the structure of plain gradients and the symbols commonly recognized by LLMs. Our experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods, improving compression rates by 10\% up to 17.2\% across various datasets and architectures. Additionally, our approach shows promising compatibility with lossy compression techniques such as quantization and sparsification. These findings highlight the significant potential of LLMs as a model for effectively handling gradients. We will release the source code upon publication.

Title: Machine Learning-based vs Deep Learning-based Anomaly Detection in Multivariate Time Series for Spacecraft Attitude Sensors

• Authors: R. Gallon, F. Schiemenz, A. Krstova, A. Menicucci, E. Gill
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17841
• Pdf URL: https://arxiv.org/pdf/2409.17841
• Copy Paste: [[2409.17841]] Machine Learning-based vs Deep Learning-based Anomaly Detection in Multivariate Time Series for Spacecraft Attitude Sensors(https://arxiv.org/abs/2409.17841)
• Keywords: interpretability
• Abstract: In the framework of Failure Detection, Isolation and Recovery (FDIR) on spacecraft, new AI-based approaches are emerging in the state of the art to overcome the limitations commonly imposed by traditional threshold checking. The present research aims at characterizing two different approaches to the problem of stuck values detection in multivariate time series coming from spacecraft attitude sensors. The analysis reveals the performance differences in the two approaches, while commenting on their interpretability and generalization to different scenarios.

Title: A New Dataset for Monocular Depth Estimation Under Viewpoint Shifts

• Authors: Aurel Pjetri (1 and 2), Stefano Caprasecca (1), Leonardo Taccari (1), Matteo Simoncini (1), Henrique Piñeiro Monteagudo (1 and 3), Walter Wallace (1), Douglas Coimbra de Andrade (4), Francesco Sambo (1), Andrew David Bagdanov (1) ((1) Verizon Connect Research, Florence, Italy, (2) Department of Information Engineering, University of Florence, Florence, Italy, (3) University of Bologna, Bologna, Italy, (4) SENAI Institute of Innovation, Rio de Janeiro, Brazil)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17851
• Pdf URL: https://arxiv.org/pdf/2409.17851
• Copy Paste: [[2409.17851]] A New Dataset for Monocular Depth Estimation Under Viewpoint Shifts(https://arxiv.org/abs/2409.17851)
• Keywords: robust
• Abstract: Monocular depth estimation is a critical task for autonomous driving and many other computer vision applications. While significant progress has been made in this field, the effects of viewpoint shifts on depth estimation models remain largely underexplored. This paper introduces a novel dataset and evaluation methodology to quantify the impact of different camera positions and orientations on monocular depth estimation performance. We propose a ground truth strategy based on homography estimation and object detection, eliminating the need for expensive lidar sensors. We collect a diverse dataset of road scenes from multiple viewpoints and use it to assess the robustness of a modern depth estimation model to geometric shifts. After assessing the validity of our strategy on a public dataset, we provide valuable insights into the limitations of current models and highlight the importance of considering viewpoint variations in real-world applications.

Title: Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

• Authors: Shaobo Ma, Chao Fang, Haikuo Shao, Zhongfeng Wang
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2409.17870
• Pdf URL: https://arxiv.org/pdf/2409.17870
• Copy Paste: [[2409.17870]] Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores(https://arxiv.org/abs/2409.17870)
• Keywords: large language model
• Abstract: Large language models (LLMs) have been widely applied but face challenges in efficient inference. While quantization methods reduce computational demands, ultra-low bit quantization with arbitrary precision is hindered by limited GPU Tensor Core support and inefficient memory management, leading to suboptimal acceleration. To address these challenges, we propose a comprehensive acceleration scheme for arbitrary precision LLMs. At its core, we introduce a novel bipolar-INT data format that facilitates parallel computing and supports symmetric quantization, effectively reducing data redundancy. Building on this, we implement an arbitrary precision matrix multiplication scheme that decomposes and recovers matrices at the bit level, enabling flexible precision while maximizing GPU Tensor Core utilization. Furthermore, we develop an efficient matrix preprocessing method that optimizes data layout for subsequent computations. Finally, we design a data recovery-oriented memory management system that strategically utilizes fast shared memory, significantly enhancing kernel execution speed and minimizing memory access latency. Experimental results demonstrate our approach's effectiveness, with up to 13\times speedup in matrix multiplication compared to NVIDIA's CUTLASS. When integrated into LLMs, we achieve up to 6.7\times inference acceleration. These improvements significantly enhance LLM inference efficiency, enabling broader and more responsive applications of LLMs.

Title: ReThink: Reveal the Threat of Electromagnetic Interference on Power Inverters

• Authors: Fengchen Yang, Zihao Dan, Kaikai Pan, Chen Yan, Xiaoyu Ji, Wenyuan Xu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.17873
• Pdf URL: https://arxiv.org/pdf/2409.17873
• Copy Paste: [[2409.17873]] ReThink: Reveal the Threat of Electromagnetic Interference on Power Inverters(https://arxiv.org/abs/2409.17873)
• Keywords: security, attack
• Abstract: With the boom of renewable energy sources (RES), the number of power inverters proliferates. Power inverters are the key electronic devices that transform the direct current (DC) power from RES to the alternating current (AC) power on the grids, and their security can affect the stable operation of RES and even power grids. This paper analyzes the security of photovoltaic (PV) inverters from the aspects of internal sensors since they serve as the foundation for safe power conversion. We discover that both the embedded current sensors and voltage sensors are vulnerable to electromagnetic interference (EMI) of 1 GHz or higher, despite electromagnetic compatibility (EMC) countermeasures. Such vulnerabilities can lead to incorrect measurements and deceiving the control algorithms, and we design ReThink that could produce three types of consequences on PV inverters by emitting carefully crafted EMI, i.e., Denial of Service (DoS), damaging inverters physically or damping the power output. We successfully validate these consequences on 5 off-the-shelf PV inverters, and even in a real-world microgrid, by transmitting EMI signals at a distance of 100-150cm and a total power within 20W. Our work aims to raise awareness of the security of power electronic devices of RES, as they represent an emerging Cyber-Physical attack surface to the future RES-dominated grid. Finally, to cope with such threats, we provide hardware and software-based countermeasures.

Title: Self-Distilled Depth Refinement with Noisy Poisson Fusion

• Authors: Jiaqi Li, Yiran Wang, Jinghong Zheng, Zihao Huang, Ke Xian, Zhiguo Cao, Jianming Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17880
• Pdf URL: https://arxiv.org/pdf/2409.17880
• Copy Paste: [[2409.17880]] Self-Distilled Depth Refinement with Noisy Poisson Fusion(https://arxiv.org/abs/2409.17880)
• Keywords: robust
• Abstract: Depth refinement aims to infer high-resolution depth with fine-grained edges and details, refining low-resolution results of depth estimation models. The prevailing methods adopt tile-based manners by merging numerous patches, which lacks efficiency and produces inconsistency. Besides, prior arts suffer from fuzzy depth boundaries and limited generalizability. Analyzing the fundamental reasons for these limitations, we model depth refinement as a noisy Poisson fusion problem with local inconsistency and edge deformation noises. We propose the Self-distilled Depth Refinement (SDDR) framework to enforce robustness against the noises, which mainly consists of depth edge representation and edge-based guidance. With noisy depth predictions as input, SDDR generates low-noise depth edge representations as pseudo-labels by coarse-to-fine self-distillation. Edge-based guidance with edge-guided gradient loss and edge-based fusion loss serves as the optimization objective equivalent to Poisson fusion. When depth maps are better refined, the labels also become more noise-free. Our model can acquire strong robustness to the noises, achieving significant improvements in accuracy, edge quality, efficiency, and generalizability on five different benchmarks. Moreover, directly training another model with edge labels produced by SDDR brings improvements, suggesting that our method could help with training robust refinement models in future works.

Title: Upper-Body Pose-based Gaze Estimation for Privacy-Preserving 3D Gaze Target Detection

• Authors: Andrea Toaiari, Vittorio Murino, Marco Cristani, Cigdem Beyan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17886
• Pdf URL: https://arxiv.org/pdf/2409.17886
• Copy Paste: [[2409.17886]] Upper-Body Pose-based Gaze Estimation for Privacy-Preserving 3D Gaze Target Detection(https://arxiv.org/abs/2409.17886)
• Keywords: privacy
• Abstract: Gaze Target Detection (GTD), i.e., determining where a person is looking within a scene from an external viewpoint, is a challenging task, particularly in 3D space. Existing approaches heavily rely on analyzing the person's appearance, primarily focusing on their face to predict the gaze target. This paper presents a novel approach to tackle this problem by utilizing the person's upper-body pose and available depth maps to extract a 3D gaze direction and employing a multi-stage or an end-to-end pipeline to predict the gazed target. When predicted accurately, the human body pose can provide valuable information about the head pose, which is a good approximation of the gaze direction, as well as the position of the arms and hands, which are linked to the activity the person is performing and the objects they are likely focusing on. Consequently, in addition to performing gaze estimation in 3D, we are also able to perform GTD simultaneously. We demonstrate state-of-the-art results on the most comprehensive publicly accessible 3D gaze target detection dataset without requiring images of the person's face, thus promoting privacy preservation in various application contexts. The code is available at this https URL.

Title: EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

• Authors: Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17892
• Pdf URL: https://arxiv.org/pdf/2409.17892
• Copy Paste: [[2409.17892]] EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models(https://arxiv.org/abs/2409.17892)
• Keywords: robust, large language model
• Abstract: In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks and PolyWrite, an open-ended generation benchmark developed in this study. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability.

Title: Self-supervised Monocular Depth Estimation with Large Kernel Attention

• Authors: Xuezhi Xiang, Yao Wang, Lei Zhang, Denis Ombati, Himaloy Himu, Xiantong Zhen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17895
• Pdf URL: https://arxiv.org/pdf/2409.17895
• Copy Paste: [[2409.17895]] Self-supervised Monocular Depth Estimation with Large Kernel Attention(https://arxiv.org/abs/2409.17895)
• Keywords: transformer
• Abstract: Self-supervised monocular depth estimation has emerged as a promising approach since it does not rely on labeled training data. Most methods combine convolution and Transformer to model long-distance dependencies to estimate depth accurately. However, Transformer treats 2D image features as 1D sequences, and positional encoding somewhat mitigates the loss of spatial information between different feature blocks, tending to overlook channel features, which limit the performance of depth estimation. In this paper, we propose a self-supervised monocular depth estimation network to get finer details. Specifically, we propose a decoder based on large kernel attention, which can model long-distance dependencies without compromising the two-dimension structure of features while maintaining feature channel adaptivity. In addition, we introduce a up-sampling module to accurately recover the fine details in the depth map. Our method achieves competitive results on the KITTI dataset.

Title: Designing Short-Stage CDC-XPUFs: Balancing Reliability, Cost, and Security in IoT Devices

• Authors: Gaoxiang Li, Yu Zhuang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17902
• Pdf URL: https://arxiv.org/pdf/2409.17902
• Copy Paste: [[2409.17902]] Designing Short-Stage CDC-XPUFs: Balancing Reliability, Cost, and Security in IoT Devices(https://arxiv.org/abs/2409.17902)
• Keywords: secure, security, attack, robust
• Abstract: The rapid expansion of Internet of Things (IoT) devices demands robust and resource-efficient security solutions. Physically Unclonable Functions (PUFs), which generate unique cryptographic keys from inherent hardware variations, offer a promising approach. However, traditional PUFs like Arbiter PUFs (APUFs) and XOR Arbiter PUFs (XOR-PUFs) are susceptible to machine learning (ML) and reliability-based attacks. In this study, we investigate Component-Differentially Challenged XOR-PUFs (CDC-XPUFs), a less explored variant, to address these vulnerabilities. We propose an optimized CDC-XPUF design that incorporates a pre-selection strategy to enhance reliability and introduces a novel lightweight architecture to reduce hardware overhead. Rigorous testing demonstrates that our design significantly lowers resource consumption, maintains strong resistance to ML attacks, and improves reliability, effectively mitigating reliability-based attacks. These results highlight the potential of CDC-XPUFs as a secure and efficient candidate for widespread deployment in resource-constrained IoT systems.

Title: Graph Reasoning with Large Language Models via Pseudo-code Prompting

• Authors: Konstantinos Skianis, Giannis Nikolentzos, Michalis Vazirgiannis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17906
• Pdf URL: https://arxiv.org/pdf/2409.17906
• Copy Paste: [[2409.17906]] Graph Reasoning with Large Language Models via Pseudo-code Prompting(https://arxiv.org/abs/2409.17906)
• Keywords: large language model
• Abstract: Large language models (LLMs) have recently achieved remarkable success in various reasoning tasks in the field of natural language processing. This success of LLMs has also motivated their use in graph-related tasks. Among others, recent work has explored whether LLMs can solve graph problems such as counting the number of connected components of a graph or computing the shortest path distance between two nodes. Although LLMs possess preliminary graph reasoning abilities, they might still struggle to solve some seemingly simple problems. In this paper, we investigate whether prompting via pseudo-code instructions can improve the performance of LLMs in solving graph problems. Our experiments demonstrate that using pseudo-code instructions generally improves the performance of all considered LLMs. The graphs, pseudo-code prompts, and evaluation code are publicly available.

Title: LKA-ReID:Vehicle Re-Identification with Large Kernel Attention

• Authors: Xuezhi Xiang, Zhushan Ma, Lei Zhang, Denis Ombati, Himaloy Himu, Xiantong Zhen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17908
• Pdf URL: https://arxiv.org/pdf/2409.17908
• Copy Paste: [[2409.17908]] LKA-ReID:Vehicle Re-Identification with Large Kernel Attention(https://arxiv.org/abs/2409.17908)
• Keywords: segmentation
• Abstract: With the rapid development of intelligent transportation systems and the popularity of smart city infrastructure, Vehicle Re-ID technology has become an important research field. The vehicle Re-ID task faces an important challenge, which is the high similarity between different vehicles. Existing methods use additional detection or segmentation models to extract differentiated local features. However, these methods either rely on additional annotations or greatly increase the computational cost. Using attention mechanism to capture global and local features is crucial to solve the challenge of high similarity between classes in vehicle Re-ID tasks. In this paper, we propose LKA-ReID with large kernel attention. Specifically, the large kernel attention (LKA) utilizes the advantages of self-attention and also benefits from the advantages of convolution, which can extract the global and local features of the vehicle more comprehensively. We also introduce hybrid channel attention (HCA) combines channel attention with spatial information, so that the model can better focus on channels and feature regions, and ignore background and other disturbing information. Experiments on VeRi-776 dataset demonstrated the effectiveness of LKA-ReID, with mAP reaches 86.65% and Rank-1 reaches 98.03%.

Title: Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect

• Authors: Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, Michalis Vazirgiannis, Eric Xing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17912
• Pdf URL: https://arxiv.org/pdf/2409.17912
• Copy Paste: [[2409.17912]] Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect(https://arxiv.org/abs/2409.17912)
• Keywords: generative, large language model
• Abstract: We introduce Atlas-Chat, the first-ever collection of large language models specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also known as Darija, we construct our instruction dataset by consolidating existing Darija language resources, creating novel datasets both manually and synthetically, and translating English instructions with stringent quality control. Atlas-Chat-9B and 2B models, fine-tuned on the dataset, exhibit superior ability in following Darija instructions and performing standard NLP tasks. Notably, our models outperform both state-of-the-art and Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., achieving a 13% performance boost over a larger 13B model on DarijaMMLU, in our newly introduced evaluation suite for Darija covering both discriminative and generative tasks. Furthermore, we perform an experimental analysis of various fine-tuning strategies and base model choices to determine optimal configurations. All our resources are publicly accessible, and we believe our work offers comprehensive design methodologies of instruction-tuning for low-resource language variants, which are often neglected in favor of data-rich languages by contemporary LLMs.

Title: WaSt-3D: Wasserstein-2 Distance for Scene-to-Scene Stylization on 3D Gaussians

• Authors: Dmytro Kotovenko, Olga Grebenkova, Nikolaos Sarafianos, Avinash Paliwal, Pingchuan Ma, Omid Poursaeed, Sreyas Mohan, Yuchen Fan, Yilei Li, Rakesh Ranjan, Björn Ommer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17917
• Pdf URL: https://arxiv.org/pdf/2409.17917
• Copy Paste: [[2409.17917]] WaSt-3D: Wasserstein-2 Distance for Scene-to-Scene Stylization on 3D Gaussians(https://arxiv.org/abs/2409.17917)
• Keywords: generative
• Abstract: While style transfer techniques have been well-developed for 2D image stylization, the extension of these methods to 3D scenes remains relatively unexplored. Existing approaches demonstrate proficiency in transferring colors and textures but often struggle with replicating the geometry of the scenes. In our work, we leverage an explicit Gaussian Splatting (GS) representation and directly match the distributions of Gaussians between style and content scenes using the Earth Mover's Distance (EMD). By employing the entropy-regularized Wasserstein-2 distance, we ensure that the transformation maintains spatial smoothness. Additionally, we decompose the scene stylization problem into smaller chunks to enhance efficiency. This paradigm shift reframes stylization from a pure generative process driven by latent space losses to an explicit matching of distributions between two Gaussian representations. Our method achieves high-resolution 3D stylization by faithfully transferring details from 3D style scenes onto the content scene. Furthermore, WaSt-3D consistently delivers results across diverse content and style scenes without necessitating any training, as it relies solely on optimization-based techniques. See our project page for additional results and source code: $\href{this https URL}{this https URL}$.

Title: Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generation

• Authors: Qihan Huang, Siming Fu, Jinlong Liu, Hao Jiang, Yipeng Yu, Jie Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17920
• Pdf URL: https://arxiv.org/pdf/2409.17920
• Copy Paste: [[2409.17920]] Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generation(https://arxiv.org/abs/2409.17920)
• Keywords: diffusion
• Abstract: Personalized text-to-image generation methods can generate customized images based on the reference images, which have garnered wide research interest. Recent methods propose a finetuning-free approach with a decoupled cross-attention mechanism to generate personalized images requiring no test-time finetuning. However, when multiple reference images are provided, the current decoupled cross-attention mechanism encounters the object confusion problem and fails to map each reference image to its corresponding object, thereby seriously limiting its scope of application. To address the object confusion problem, in this work we investigate the relevance of different positions of the latent image features to the target object in diffusion model, and accordingly propose a weighted-merge method to merge multiple reference image features into the corresponding objects. Next, we integrate this weighted-merge method into existing pre-trained models and continue to train the model on a multi-object dataset constructed from the open-sourced SA-1B dataset. To mitigate object confusion and reduce training costs, we propose an object quality score to estimate the image quality for the selection of high-quality training samples. Furthermore, our weighted-merge training framework can be employed on single-object generation when a single object has multiple reference images. The experiments verify that our method achieves superior performance to the state-of-the-arts on the Concept101 dataset and DreamBooth dataset of multi-object personalized image generation, and remarkably improves the performance on single-object personalized image generation. Our code is available at this https URL.

Title: Pioneering Reliable Assessment in Text-to-Image Knowledge Editing: Leveraging a Fine-Grained Dataset and an Innovative Criterion

• Authors: Hengrui Gu, Kaixiong Zhou, Yili Wang, Ruobing Wang, Xin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.17928
• Pdf URL: https://arxiv.org/pdf/2409.17928
• Copy Paste: [[2409.17928]] Pioneering Reliable Assessment in Text-to-Image Knowledge Editing: Leveraging a Fine-Grained Dataset and an Innovative Criterion(https://arxiv.org/abs/2409.17928)
• Keywords: diffusion
• Abstract: During pre-training, the Text-to-Image (T2I) diffusion models encode factual knowledge into their parameters. These parameterized facts enable realistic image generation, but they may become obsolete over time, thereby misrepresenting the current state of the world. Knowledge editing techniques aim to update model knowledge in a targeted way. However, facing the dual challenges posed by inadequate editing datasets and unreliable evaluation criterion, the development of T2I knowledge editing encounter difficulties in effectively generalizing injected knowledge. In this work, we design a T2I knowledge editing framework by comprehensively spanning on three phases: First, we curate a dataset \textbf{CAKE}, comprising paraphrase and multi-object test, to enable more fine-grained assessment on knowledge generalization. Second, we propose a novel criterion, \textbf{adaptive CLIP threshold}, to effectively filter out false successful images under the current criterion and achieve reliable editing evaluation. Finally, we introduce \textbf{MPE}, a simple but effective approach for T2I knowledge editing. Instead of tuning parameters, MPE precisely recognizes and edits the outdated part of the conditioning text-prompt to accommodate the up-to-date knowledge. A straightforward implementation of MPE (Based on in-context learning) exhibits better overall performance than previous model editors. We hope these efforts can further promote faithful evaluation of T2I knowledge editing methods.

Title: The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification

• Authors: Andreas Waldis, Joel Birrer, Anne Lauscher, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17929
• Pdf URL: https://arxiv.org/pdf/2409.17929
• Copy Paste: [[2409.17929]] The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification(https://arxiv.org/abs/2409.17929)
• Keywords: fair
• Abstract: Gender-fair language, an evolving German linguistic variation, fosters inclusion by addressing all genders or using neutral forms. Nevertheless, there is a significant lack of resources to assess the impact of this linguistic shift on classification using language models (LMs), which are probably not trained on such variations. To address this gap, we present Lou, the first dataset featuring high-quality reformulations for German text classification covering seven tasks, like stance detection and toxicity classification. Evaluating 16 mono- and multi-lingual LMs on Lou shows that gender-fair language substantially impacts predictions by flipping labels, reducing certainty, and altering attention patterns. However, existing evaluations remain valid, as LM rankings of original and reformulated instances do not significantly differ. While we offer initial insights on the effect on German text classification, the findings likely apply to other languages, as consistent patterns were observed in multi-lingual and English LMs.

Title: Adaptive Stream Processing on Edge Devices through Active Inference

• Authors: Boris Sedlak, Victor Casamayor Pujol, Andrea Morichetta, Praveen Kumar Donta, Schahram Dustdar
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.17937
• Pdf URL: https://arxiv.org/pdf/2409.17937
• Copy Paste: [[2409.17937]] Adaptive Stream Processing on Edge Devices through Active Inference(https://arxiv.org/abs/2409.17937)
• Keywords: privacy
• Abstract: The current scenario of IoT is witnessing a constant increase on the volume of data, which is generated in constant stream, calling for novel architectural and logical solutions for processing it. Moving the data handling towards the edge of the computing spectrum guarantees better distribution of load and, in principle, lower latency and better privacy. However, managing such a structure is complex, especially when requirements, also referred to Service Level Objectives (SLOs), specified by applications' owners and infrastructure managers need to be ensured. Despite the rich number of proposals of Machine Learning (ML) based management solutions, researchers and practitioners yet struggle to guarantee long-term prediction and control, and accurate troubleshooting. Therefore, we present a novel ML paradigm based on Active Inference (AIF) -- a concept from neuroscience that describes how the brain constantly predicts and evaluates sensory information to decrease long-term surprise. We implement it and evaluate it in a heterogeneous real stream processing use case, where an AIF-based agent continuously optimizes the fulfillment of three SLOs for three autonomous driving services running on multiple devices. The agent used causal knowledge to gradually develop an understanding of how its actions are related to requirements fulfillment, and which configurations to favor. Through this approach, our agent requires up to thirty iterations to converge to the optimal solution, showing the capability of offering accurate results in a short amount of time. Furthermore, thanks to AIF and its causal structures, our method guarantees full transparency on the decision making, making the interpretation of the results and the troubleshooting effortless.

Title: Perturb, Attend, Detect and Localize (PADL): Robust Proactive Image Defense

• Authors: Filippo Bartolucci, Iacopo Masi, Giuseppe Lisanti
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17941
• Pdf URL: https://arxiv.org/pdf/2409.17941
• Copy Paste: [[2409.17941]] Perturb, Attend, Detect and Localize (PADL): Robust Proactive Image Defense(https://arxiv.org/abs/2409.17941)
• Keywords: protect, defense, attack, robust, fair, diffusion, generative
• Abstract: Image manipulation detection and localization have received considerable attention from the research community given the blooming of Generative Models (GMs). Detection methods that follow a passive approach may overfit to specific GMs, limiting their application in real-world scenarios, due to the growing diversity of generative models. Recently, approaches based on a proactive framework have shown the possibility of dealing with this limitation. However, these methods suffer from two main limitations, which raises concerns about potential vulnerabilities: i) the manipulation detector is not robust to noise and hence can be easily fooled; ii) the fact that they rely on fixed perturbations for image protection offers a predictable exploit for malicious attackers, enabling them to reverse-engineer and evade detection. To overcome this issue we propose PADL, a new solution able to generate image-specific perturbations using a symmetric scheme of encoding and decoding based on cross-attention, which drastically reduces the possibility of reverse engineering, even when evaluated with adaptive attack [31]. Additionally, PADL is able to pinpoint manipulated areas, facilitating the identification of specific regions that have undergone alterations, and has more generalization power than prior art on held-out generative models. Indeed, although being trained only on an attribute manipulation GAN model [15], our method generalizes to a range of unseen models with diverse architectural designs, such as StarGANv2, BlendGAN, DiffAE, StableDiffusion and StableDiffusionXL. Additionally, we introduce a novel evaluation protocol, which offers a fair evaluation of localisation performance in function of detection accuracy and better captures real-world scenarios.

Title: Weak-To-Strong Backdoor Attacks for LLMs with Contrastive Knowledge Distillation

• Authors: Shuai Zhao, Leilei Gan, Zhongliang Guo, Xiaobao Wu, Luwei Xiao, Xiaoyu Xu, Cong-Duy Nguyen, Luu Anh Tuan
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17946
• Pdf URL: https://arxiv.org/pdf/2409.17946
• Copy Paste: [[2409.17946]] Weak-To-Strong Backdoor Attacks for LLMs with Contrastive Knowledge Distillation(https://arxiv.org/abs/2409.17946)
• Keywords: attack, large language model
• Abstract: Despite being widely applied due to their exceptional capabilities, Large Language Models (LLMs) have been proven to be vulnerable to backdoor attacks. These attacks introduce targeted vulnerabilities into LLMs by poisoning training samples and full-parameter fine-tuning. However, this kind of backdoor attack is limited since they require significant computational resources, especially as the size of LLMs increases. Besides, parameter-efficient fine-tuning (PEFT) offers an alternative but the restricted parameter updating may impede the alignment of triggers with target labels. In this study, we first verify that backdoor attacks with PEFT may encounter challenges in achieving feasible performance. To address these issues and improve the effectiveness of backdoor attacks with PEFT, we propose a novel backdoor attack algorithm from weak to strong based on contrastive knowledge distillation (W2SAttack). Specifically, we poison small-scale language models through full-parameter fine-tuning to serve as the teacher model. The teacher model then covertly transfers the backdoor to the large-scale student model through contrastive knowledge distillation, which employs PEFT. Theoretical analysis reveals that W2SAttack has the potential to augment the effectiveness of backdoor attacks. We demonstrate the superior performance of W2SAttack on classification tasks across four language models, four backdoor attack algorithms, and two different architectures of teacher models. Experimental results indicate success rates close to 100% for backdoor attacks targeting PEFT.

Title: Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition

• Authors: Xinpeng Yin, Wenming Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17951
• Pdf URL: https://arxiv.org/pdf/2409.17951
• Copy Paste: [[2409.17951]] Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition(https://arxiv.org/abs/2409.17951)
• Keywords: robust
• Abstract: In self-supervised skeleton-based action recognition, the mask reconstruction paradigm is gaining interest in enhancing model refinement and robustness through effective masking. However, previous works primarily relied on a single masking criterion, resulting in the model overfitting specific features and overlooking other effective information. In this paper, we introduce a hierarchy and attention guided cross-masking framework (HA-CM) that applies masking to skeleton sequences from both spatial and temporal perspectives. Specifically, in spatial graphs, we utilize hyperbolic space to maintain joint distinctions and effectively preserve the hierarchical structure of high-dimensional skeletons, employing joint hierarchy as the masking criterion. In temporal flows, we substitute traditional distance metrics with the global attention of joints for masking, addressing the convergence of distances in high-dimensional space and the lack of a global perspective. Additionally, we incorporate cross-contrast loss based on the cross-masking framework into the loss function to enhance the model's learning of instance-level features. HA-CM shows efficiency and universality on three public large-scale datasets, NTU-60, NTU-120, and PKU-MMD. The source code of our HA-CM is available at this https URL.

Title: The Hard Positive Truth about Vision-Language Compositionality

• Authors: Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, Ranjay Krishna
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17958
• Pdf URL: https://arxiv.org/pdf/2409.17958
• Copy Paste: [[2409.17958]] The Hard Positive Truth about Vision-Language Compositionality(https://arxiv.org/abs/2409.17958)
• Keywords: robust
• Abstract: Several benchmarks have concluded that our best vision-language models (e.g., CLIP) are lacking in compositionality. Given an image, these benchmarks probe a model's ability to identify its associated caption amongst a set of compositional distractors. In response, a surge of recent proposals show improvements by finetuning CLIP with distractors as hard negatives. Our investigations reveal that these improvements have, in fact, been significantly overstated -- because existing benchmarks do not probe whether finetuned vision-language models remain invariant to hard positives. By curating an evaluation dataset with 112,382 hard negatives and hard positives, we uncover that including hard positives decreases CLIP's performance by 12.9%, while humans perform effortlessly at 99%. CLIP finetuned with hard negatives results in an even larger decrease, up to 38.7%. With this finding, we then produce a 1,775,259 image-text training set with both hard negative and hard positive captions. By training with both, we see improvements on existing benchmarks while simultaneously improving performance on hard positives, indicating a more robust improvement in compositionality. Our work suggests the need for future research to rigorously test and improve CLIP's understanding of semantic relationships between related "positive" concepts.

Title: CNCA: Toward Customizable and Natural Generation of Adversarial Camouflage for Vehicle Detectors

• Authors: Linye Lyu, Jiawei Zhou, Daojing He, Yu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17963
• Pdf URL: https://arxiv.org/pdf/2409.17963
• Copy Paste: [[2409.17963]] CNCA: Toward Customizable and Natural Generation of Adversarial Camouflage for Vehicle Detectors(https://arxiv.org/abs/2409.17963)
• Keywords: attack, robust, diffusion
• Abstract: Prior works on physical adversarial camouflage against vehicle detectors mainly focus on the effectiveness and robustness of the attack. The current most successful methods optimize 3D vehicle texture at a pixel level. However, this results in conspicuous and attention-grabbing patterns in the generated camouflage, which humans can easily identify. To address this issue, we propose a Customizable and Natural Camouflage Attack (CNCA) method by leveraging an off-the-shelf pre-trained diffusion model. By sampling the optimal texture image from the diffusion model with a user-specific text prompt, our method can generate natural and customizable adversarial camouflage while maintaining high attack performance. With extensive experiments on the digital and physical worlds and user studies, the results demonstrate that our proposed method can generate significantly more natural-looking camouflage than the state-of-the-art baselines while achieving competitive attack performance. Our code is available at \href{this https URL}{this https URL}

Title: BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Search

• Authors: Linzhuang Sun, Hao Liang, Wentao Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17972
• Pdf URL: https://arxiv.org/pdf/2409.17972
• Copy Paste: [[2409.17972]] BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Search(https://arxiv.org/abs/2409.17972)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have exhibited exceptional performance across a broad range of tasks and domains. However, they still encounter difficulties in solving mathematical problems due to the rigorous and logical nature of mathematics. Previous studies have employed techniques such as supervised fine-tuning (SFT), prompt engineering, and search-based methods to improve the mathematical problem-solving abilities of LLMs. Despite these efforts, their performance remains suboptimal and demands substantial computational resources. To address this issue, we propose a novel approach, BEATS, to enhance mathematical problem-solving abilities. Our method leverages newly designed prompts that guide the model to iteratively rewrite, advance by one step, and generate answers based on previous steps. Additionally, we introduce a new back-verification technique that uses LLMs to validate the correctness of the generated answers. Furthermore, we employ a pruning tree search to optimize search time while achieving strong performance. Notably, our method improves Qwen2-7b-Instruct's score from 36.94 to 61.52, outperforming GPT4's 42.5 on the MATH benchmark.

Title: Cross-Modality Attack Boosted by Gradient-Evolutionary Multiform Optimization

• Authors: Yunpeng Gong, Qingyuan Zeng, Dejun Xu, Zhenzhong Wang, Min Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17977
• Pdf URL: https://arxiv.org/pdf/2409.17977
• Copy Paste: [[2409.17977]] Cross-Modality Attack Boosted by Gradient-Evolutionary Multiform Optimization(https://arxiv.org/abs/2409.17977)
• Keywords: security, attack, robust
• Abstract: In recent years, despite significant advancements in adversarial attack research, the security challenges in cross-modal scenarios, such as the transferability of adversarial attacks between infrared, thermal, and RGB images, have been overlooked. These heterogeneous image modalities collected by different hardware devices are widely prevalent in practical applications, and the substantial differences between modalities pose significant challenges to attack transferability. In this work, we explore a novel cross-modal adversarial attack strategy, termed multiform attack. We propose a dual-layer optimization framework based on gradient-evolution, facilitating efficient perturbation transfer between modalities. In the first layer of optimization, the framework utilizes image gradients to learn universal perturbations within each modality and employs evolutionary algorithms to search for shared perturbations with transferability across different modalities through secondary optimization. Through extensive testing on multiple heterogeneous datasets, we demonstrate the superiority and robustness of Multiform Attack compared to existing techniques. This work not only enhances the transferability of cross-modal adversarial attacks but also provides a new perspective for understanding security vulnerabilities in cross-modal systems.

Title: HydraViT: Stacking Heads for a Scalable ViT

• Authors: Janek Haberer, Ali Hojjat, Olaf Landsiedel
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17978
• Pdf URL: https://arxiv.org/pdf/2409.17978
• Copy Paste: [[2409.17978]] HydraViT: Stacking Heads for a Scalable ViT(https://arxiv.org/abs/2409.17978)
• Keywords: transformer
• Abstract: The architecture of Vision Transformers (ViTs), particularly the Multi-head Attention (MHA) mechanism, imposes substantial hardware demands. Deploying ViTs on devices with varying constraints, such as mobile phones, requires multiple models of different sizes. However, this approach has limitations, such as training and storing each required model separately. This paper introduces HydraViT, a novel approach that addresses these limitations by stacking attention heads to achieve a scalable ViT. By repeatedly changing the size of the embedded dimensions throughout each layer and their corresponding number of attention heads in MHA during training, HydraViT induces multiple subnetworks. Thereby, HydraViT achieves adaptability across a wide spectrum of hardware environments while maintaining performance. Our experimental results demonstrate the efficacy of HydraViT in achieving a scalable ViT with up to 10 subnetworks, covering a wide range of resource constraints. HydraViT achieves up to 5 p.p. more accuracy with the same GMACs and up to 7 p.p. more accuracy with the same throughput on ImageNet-1K compared to the baselines, making it an effective solution for scenarios where hardware availability is diverse or varies over time. Source code available at this https URL.

Title: Supra-Laplacian Encoding for Transformer on Dynamic Graphs

• Authors: Yannis Karmim, Marc Lafon, Raphaël Fournier S'niehotta, Nicolas Thome
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17986
• Pdf URL: https://arxiv.org/pdf/2409.17986
• Copy Paste: [[2409.17986]] Supra-Laplacian Encoding for Transformer on Dynamic Graphs(https://arxiv.org/abs/2409.17986)
• Keywords: transformer
• Abstract: Fully connected Graph Transformers (GT) have rapidly become prominent in the static graph community as an alternative to Message-Passing models, which suffer from a lack of expressivity, oversquashing, and under-reaching. However, in a dynamic context, by interconnecting all nodes at multiple snapshots with self-attention, GT loose both structural and temporal information. In this work, we introduce Supra-LAplacian encoding for spatio-temporal TransformErs (SLATE), a new spatio-temporal encoding to leverage the GT architecture while keeping spatio-temporal information. Specifically, we transform Discrete Time Dynamic Graphs into multi-layer graphs and take advantage of the spectral properties of their associated supra-Laplacian matrix. Our second contribution explicitly model nodes' pairwise relationships with a cross-attention mechanism, providing an accurate edge representation for dynamic link prediction. SLATE outperforms numerous state-of-the-art methods based on Message-Passing Graph Neural Networks combined with recurrent models (e.g LSTM), and Dynamic Graph Transformers, on 9 datasets. Code and instructions to reproduce our results will be open-sourced.

Title: LLM4Brain: Training a Large Language Model for Brain Video Understanding

• Authors: Ruizhe Zheng, Lichao Sun
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.17987
• Pdf URL: https://arxiv.org/pdf/2409.17987
• Copy Paste: [[2409.17987]] LLM4Brain: Training a Large Language Model for Brain Video Understanding(https://arxiv.org/abs/2409.17987)
• Keywords: large language model
• Abstract: Decoding visual-semantic information from brain signals, such as functional MRI (fMRI), across different subjects poses significant challenges, including low signal-to-noise ratio, limited data availability, and cross-subject variability. Recent advancements in large language models (LLMs) show remarkable effectiveness in processing multimodal information. In this study, we introduce an LLM-based approach for reconstructing visual-semantic information from fMRI signals elicited by video stimuli. Specifically, we employ fine-tuning techniques on an fMRI encoder equipped with adaptors to transform brain responses into latent representations aligned with the video stimuli. Subsequently, these representations are mapped to textual modality by LLM. In particular, we integrate self-supervised domain adaptation methods to enhance the alignment between visual-semantic information and brain responses. Our proposed method achieves good results using various quantitative semantic metrics, while yielding similarity with ground-truth information.

Title: InterNet: Unsupervised Cross-modal Homography Estimation Based on Interleaved Modality Transfer and Self-supervised Homography Prediction

• Authors: Junchen Yu, Si-Yuan Cao, Runmin Zhang, Chenghao Zhang, Jianxin Hu, Zhu Yu, Hui-liang Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.17993
• Pdf URL: https://arxiv.org/pdf/2409.17993
• Copy Paste: [[2409.17993]] InterNet: Unsupervised Cross-modal Homography Estimation Based on Interleaved Modality Transfer and Self-supervised Homography Prediction(https://arxiv.org/abs/2409.17993)
• Keywords: robust
• Abstract: We propose a novel unsupervised cross-modal homography estimation framework, based on interleaved modality transfer and self-supervised homography prediction, named InterNet. InterNet integrates modality transfer and self-supervised homography estimation, introducing an innovative interleaved optimization framework to alternately promote both components. The modality transfer gradually narrows the modality gaps, facilitating the self-supervised homography estimation to fully leverage the synthetic intra-modal data. The self-supervised homography estimation progressively achieves reliable predictions, thereby providing robust cross-modal supervision for the modality transfer. To further boost the estimation accuracy, we also formulate a fine-grained homography feature loss to improve the connection between two components. Furthermore, we employ a simple yet effective distillation training technique to reduce model parameters and improve cross-domain generalization ability while maintaining comparable performance. Experiments reveal that InterNet achieves the state-of-the-art (SOTA) performance among unsupervised methods, and even outperforms many supervised methods such as MHN and LocalTrans.

Title: Multilingual Evaluation of Long Context Retrieval and Reasoning

• Authors: Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel, Russell Scheinberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18006
• Pdf URL: https://arxiv.org/pdf/2409.18006
• Copy Paste: [[2409.18006]] Multilingual Evaluation of Long Context Retrieval and Reasoning(https://arxiv.org/abs/2409.18006)
• Keywords: large language model
• Abstract: Recent large language models (LLMs) demonstrate impressive capabilities in handling long contexts, some exhibiting near-perfect recall on synthetic retrieval tasks. However, these evaluations have mainly focused on English text and involved a single target sentence within lengthy contexts. Our work investigates how LLM performance generalizes to multilingual settings with multiple hidden target sentences. We comprehensively evaluate several long-context LLMs on retrieval and reasoning tasks across five languages: English, Vietnamese, Indonesian, Swahili, and Somali. These languages share the Latin script but belong to distinct language families and resource levels. Our analysis reveals a significant performance gap between languages. The best-performing models such as Gemini-1.5 and GPT-4o, achieve around 96% accuracy in English to around 36% in Somali with a single target sentence. However, this accuracy drops to 40% in English and 0% in Somali when dealing with three target sentences. Our findings highlight the challenges long-context LLMs face when processing longer contexts, an increase in the number of target sentences, or languages of lower resource levels.

Title: Transferring disentangled representations: bridging the gap between synthetic and real images

• Authors: Jacopo Dapueto, Nicoletta Noceti, Francesca Odone
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18017
• Pdf URL: https://arxiv.org/pdf/2409.18017
• Copy Paste: [[2409.18017]] Transferring disentangled representations: bridging the gap between synthetic and real images(https://arxiv.org/abs/2409.18017)
• Keywords: generative
• Abstract: Developing meaningful and efficient representations that separate the fundamental structure of the data generation mechanism is crucial in representation learning. However, Disentangled Representation Learning has not fully shown its potential on real images, because of correlated generative factors, their resolution and limited access to ground truth labels. Specifically on the latter, we investigate the possibility of leveraging synthetic data to learn general-purpose disentangled representations applicable to real data, discussing the effect of fine-tuning and what properties of disentanglement are preserved after the transfer. We provide an extensive empirical study to address these issues. In addition, we propose a new interpretable intervention-based metric, to measure the quality of factors encoding in the representation. Our results indicate that some level of disentanglement, transferring a representation from synthetic to real data, is possible and effective.

Title: DARE: Diverse Visual Question Answering with Robustness Evaluation

• Authors: Hannah Sterz, Jonas Pfeiffer, Ivan Vulić
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18023
• Pdf URL: https://arxiv.org/pdf/2409.18023
• Copy Paste: [[2409.18023]] DARE: Diverse Visual Question Answering with Robustness Evaluation(https://arxiv.org/abs/2409.18023)
• Keywords: robust, large language model
• Abstract: Vision Language Models (VLMs) extend remarkable capabilities of text-only large language models and vision-only models, and are able to learn from and process multi-modal vision-text input. While modern VLMs perform well on a number of standard image classification and image-text matching tasks, they still struggle with a number of crucial vision-language (VL) reasoning abilities such as counting and spatial reasoning. Moreover, while they might be very brittle to small variations in instructions and/or evaluation protocols, existing benchmarks fail to evaluate their robustness (or rather the lack of it). In order to couple challenging VL scenarios with comprehensive robustness evaluation, we introduce DARE, Diverse Visual Question Answering with Robustness Evaluation, a carefully created and curated multiple-choice VQA benchmark. DARE evaluates VLM performance on five diverse categories and includes four robustness-oriented evaluations based on the variations of: prompts, the subsets of answer options, the output format and the number of correct answers. Among a spectrum of other findings, we report that state-of-the-art VLMs still struggle with questions in most categories and are unable to consistently deliver their peak performance across the tested robustness evaluations. The worst case performance across the subsets of options is up to 34% below the performance in the standard case. The robustness of the open-source VLMs such as LLaVA 1.6 and Idefics2 cannot match the closed-source models such as GPT-4 and Gemini, but even the latter remain very brittle to different variations.

Title: An Adversarial Perspective on Machine Unlearning for AI Safety

• Authors: Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, Javier Rando
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18025
• Pdf URL: https://arxiv.org/pdf/2409.18025
• Copy Paste: [[2409.18025]] An Adversarial Perspective on Machine Unlearning for AI Safety(https://arxiv.org/abs/2409.18025)
• Keywords: protect, robust, large language model
• Abstract: Large language models are finetuned to refuse questions about hazardous knowledge, but these protections can often be bypassed. Unlearning methods aim at completely removing hazardous capabilities from models and make them inaccessible to adversaries. This work challenges the fundamental differences between unlearning and traditional safety post-training from an adversarial perspective. We demonstrate that existing jailbreak methods, previously reported as ineffective against unlearning, can be successful when applied carefully. Furthermore, we develop a variety of adaptive methods that recover most supposedly unlearned capabilities. For instance, we show that finetuning on 10 unrelated examples or removing specific directions in the activation space can recover most hazardous capabilities for models edited with RMU, a state-of-the-art unlearning method. Our findings challenge the robustness of current unlearning approaches and question their advantages over safety training.

Title: ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learning

• Authors: Song Wang, Zhongdao Wang, Jiawei Yu, Wentong Li, Bailan Feng, Junbo Chen, Jianke Zhu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.18026
• Pdf URL: https://arxiv.org/pdf/2409.18026
• Copy Paste: [[2409.18026]] ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learning(https://arxiv.org/abs/2409.18026)
• Keywords: robust
• Abstract: Vision-centric semantic occupancy prediction plays a crucial role in autonomous driving, which requires accurate and reliable predictions from low-cost sensors. Although having notably narrowed the accuracy gap with LiDAR, there is still few research effort to explore the reliability in predicting semantic occupancy from camera. In this paper, we conduct a comprehensive evaluation of existing semantic occupancy prediction models from a reliability perspective for the first time. Despite the gradual alignment of camera-based models with LiDAR in term of accuracy, a significant reliability gap persists. To addresses this concern, we propose ReliOcc, a method designed to enhance the reliability of camera-based occupancy networks. ReliOcc provides a plug-and-play scheme for existing models, which integrates hybrid uncertainty from individual voxels with sampling-based noise and relative voxels through mix-up learning. Besides, an uncertainty-aware calibration strategy is devised to further enhance model reliability in offline mode. Extensive experiments under various settings demonstrate that ReliOcc significantly enhances model reliability while maintaining the accuracy of both geometric and semantic predictions. Importantly, our proposed approach exhibits robustness to sensor failures and out of domain noises during inference.

Title: EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

• Authors: Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, Dingdong Wang, Kun Xiang, Haoyuan Li, Haoli Bai, Jianhua Han, Xiaohui Li, Weike Jin, Nian Xie, Yu Zhang, James T. Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang, Qun Liu, Lanqing Hong, Lu Hou, Hang Xu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18042
• Pdf URL: https://arxiv.org/pdf/2409.18042
• Copy Paste: [[2409.18042]] EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions(https://arxiv.org/abs/2409.18042)
• Keywords: large language model
• Abstract: GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language Models to perceive and generate images, texts, and speeches end-to-end with publicly available data remains challenging in the open-source community. Existing vision-language models rely on external tools for the speech processing, while speech-language models still suffer from limited or even without vision-understanding abilities. To address this gap, we propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large Language Models with end-to-end speech capabilities while maintaining the leading vision-language performance. With a semantic-acoustic disentangled speech tokenizer, we notice surprisingly that omni-modal alignment can further enhance vision-language and speech abilities compared with the corresponding bi-modal aligned counterparts. Moreover, a lightweight style module is proposed for flexible speech style controls (e.g., emotions and pitches). For the first time, EMOVA achieves state-of-the-art performance on both the vision-language and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue with vivid emotions.

Title: Revisit Anything: Visual Place Recognition via Image Segment Retrieval

• Authors: Kartik Garg, Sai Shubodh Puligilla, Shishir Kolathaya, Madhava Krishna, Sourav Garg
• Subjects: cs.CV, cs.AI, cs.IR, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.18049
• Pdf URL: https://arxiv.org/pdf/2409.18049
• Copy Paste: [[2409.18049]] Revisit Anything: Visual Place Recognition via Image Segment Retrieval(https://arxiv.org/abs/2409.18049)
• Keywords: segmentation
• Abstract: Accurately recognizing a revisited place is crucial for embodied agents to localize and navigate. This requires visual representations to be distinct, despite strong variations in camera viewpoint and scene appearance. Existing visual place recognition pipelines encode the "whole" image and search for matches. This poses a fundamental challenge in matching two images of the same place captured from different camera viewpoints: "the similarity of what overlaps can be dominated by the dissimilarity of what does not overlap". We address this by encoding and searching for "image segments" instead of the whole images. We propose to use open-set image segmentation to decompose an image into `meaningful' entities (i.e., things and stuff). This enables us to create a novel image representation as a collection of multiple overlapping subgraphs connecting a segment with its neighboring segments, dubbed SuperSegment. Furthermore, to efficiently encode these SuperSegments into compact vector representations, we propose a novel factorized representation of feature aggregation. We show that retrieving these partial representations leads to significantly higher recognition recall than the typical whole image based retrieval. Our segments-based approach, dubbed SegVLAD, sets a new state-of-the-art in place recognition on a diverse selection of benchmark datasets, while being applicable to both generic and task-specialized image encoders. Finally, we demonstrate the potential of our method to ``revisit anything'' by evaluating our method on an object instance retrieval task, which bridges the two disparate areas of research: visual place recognition and object-goal navigation, through their common aim of recognizing goal objects specific to a place. Source code: this https URL.

Title: Stable Video Portraits

• Authors: Mirela Ostrek, Justus Thies
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18083
• Pdf URL: https://arxiv.org/pdf/2409.18083
• Copy Paste: [[2409.18083]] Stable Video Portraits(https://arxiv.org/abs/2409.18083)
• Keywords: diffusion, generative
• Abstract: Rapid advances in the field of generative AI and text-to-image methods in particular have transformed the way we interact with and perceive computer-generated imagery today. In parallel, much progress has been made in 3D face reconstruction, using 3D Morphable Models (3DMM). In this paper, we present SVP, a novel hybrid 2D/3D generation method that outputs photorealistic videos of talking faces leveraging a large pre-trained text-to-image prior (2D), controlled via a 3DMM (3D). Specifically, we introduce a person-specific fine-tuning of a general 2D stable diffusion model which we lift to a video model by providing temporal 3DMM sequences as conditioning and by introducing a temporal denoising procedure. As an output, this model generates temporally smooth imagery of a person with 3DMM-based controls, i.e., a person-specific avatar. The facial appearance of this person-specific avatar can be edited and morphed to text-defined celebrities, without any fine-tuning at test time. The method is analyzed quantitatively and qualitatively, and we show that our method outperforms state-of-the-art monocular head avatar methods.

Title: DiffSSC: Semantic LiDAR Scan Completion using Denoising Diffusion Probabilistic Models

• Authors: Helin Cao, Sven Behnke
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.18092
• Pdf URL: https://arxiv.org/pdf/2409.18092
• Copy Paste: [[2409.18092]] DiffSSC: Semantic LiDAR Scan Completion using Denoising Diffusion Probabilistic Models(https://arxiv.org/abs/2409.18092)
• Keywords: diffusion
• Abstract: Perception systems play a crucial role in autonomous driving, incorporating multiple sensors and corresponding computer vision algorithms. 3D LiDAR sensors are widely used to capture sparse point clouds of the vehicle's surroundings. However, such systems struggle to perceive occluded areas and gaps in the scene due to the sparsity of these point clouds and their lack of semantics. To address these challenges, Semantic Scene Completion (SSC) jointly predicts unobserved geometry and semantics in the scene given raw LiDAR measurements, aiming for a more complete scene representation. Building on promising results of diffusion models in image generation and super-resolution tasks, we propose their extension to SSC by implementing the noising and denoising diffusion processes in the point and semantic spaces individually. To control the generation, we employ semantic LiDAR point clouds as conditional input and design local and global regularization losses to stabilize the denoising process. We evaluate our approach on autonomous driving datasets and our approach outperforms the state-of-the-art for SSC.

Title: EfficientCrackNet: A Lightweight Model for Crack Segmentation

• Authors: Abid Hasan Zim, Aquib Iqbal, Zaid Al-Huda, Asad Malik, Minoru Kuribayash
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18099
• Pdf URL: https://arxiv.org/pdf/2409.18099
• Copy Paste: [[2409.18099]] EfficientCrackNet: A Lightweight Model for Crack Segmentation(https://arxiv.org/abs/2409.18099)
• Keywords: robust, extraction, transformer, segmentation
• Abstract: Crack detection, particularly from pavement images, presents a formidable challenge in the domain of computer vision due to several inherent complexities such as intensity inhomogeneity, intricate topologies, low contrast, and noisy backgrounds. Automated crack detection is crucial for maintaining the structural integrity of essential infrastructures, including buildings, pavements, and bridges. Existing lightweight methods often face challenges including computational inefficiency, complex crack patterns, and difficult backgrounds, leading to inaccurate detection and impracticality for real-world applications. To address these limitations, we propose EfficientCrackNet, a lightweight hybrid model combining Convolutional Neural Networks (CNNs) and transformers for precise crack segmentation. EfficientCrackNet integrates depthwise separable convolutions (DSC) layers and MobileViT block to capture both global and local features. The model employs an Edge Extraction Method (EEM) and for efficient crack edge detection without pretraining, and Ultra-Lightweight Subspace Attention Module (ULSAM) to enhance feature extraction. Extensive experiments on three benchmark datasets Crack500, DeepCrack, and GAPs384 demonstrate that EfficientCrackNet achieves superior performance compared to existing lightweight models, while requiring only 0.26M parameters, and 0.483 FLOPs (G). The proposed model offers an optimal balance between accuracy and computational efficiency, outperforming state-of-the-art lightweight models, and providing a robust and adaptable solution for real-world crack segmentation.

Title: Self-supervised Pretraining for Cardiovascular Magnetic Resonance Cine Segmentation

• Authors: Rob A. J. de Mooij, Josien P. W. Pluim, Cian M. Scannell
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18100
• Pdf URL: https://arxiv.org/pdf/2409.18100
• Copy Paste: [[2409.18100]] Self-supervised Pretraining for Cardiovascular Magnetic Resonance Cine Segmentation(https://arxiv.org/abs/2409.18100)
• Keywords: segmentation
• Abstract: Self-supervised pretraining (SSP) has shown promising results in learning from large unlabeled datasets and, thus, could be useful for automated cardiovascular magnetic resonance (CMR) short-axis cine segmentation. However, inconsistent reports of the benefits of SSP for segmentation have made it difficult to apply SSP to CMR. Therefore, this study aimed to evaluate SSP methods for CMR cine segmentation. To this end, short-axis cine stacks of 296 subjects (90618 2D slices) were used for unlabeled pretraining with four SSP methods; SimCLR, positional contrastive learning, DINO, and masked image modeling (MIM). Subsets of varying numbers of subjects were used for supervised fine-tuning of 2D models for each SSP method, as well as to train a 2D baseline model from scratch. The fine-tuned models were compared to the baseline using the 3D Dice similarity coefficient (DSC) in a test dataset of 140 subjects. The SSP methods showed no performance gains with the largest supervised fine-tuning subset compared to the baseline (DSC = 0.89). When only 10 subjects (231 2D slices) are available for supervised training, SSP using MIM (DSC = 0.86) improves over training from scratch (DSC = 0.82). This study found that SSP is valuable for CMR cine segmentation when labeled training data is scarce, but does not aid state-of-the-art deep learning methods when ample labeled data is available. Moreover, the choice of SSP method is important. The code is publicly available at: this https URL

Title: Find Rhinos without Finding Rhinos: Active Learning with Multimodal Imagery of South African Rhino Habitats

• Authors: Lucia Gordon, Nikhil Behari, Samuel Collier, Elizabeth Bondi-Kelly, Jackson A. Killian, Catherine Ressijac, Peter Boucher, Andrew Davies, Milind Tambe
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18104
• Pdf URL: https://arxiv.org/pdf/2409.18104
• Copy Paste: [[2409.18104]] Find Rhinos without Finding Rhinos: Active Learning with Multimodal Imagery of South African Rhino Habitats(https://arxiv.org/abs/2409.18104)
• Keywords: protect
• Abstract: Much of Earth's charismatic megafauna is endangered by human activities, particularly the rhino, which is at risk of extinction due to the poaching crisis in Africa. Monitoring rhinos' movement is crucial to their protection but has unfortunately proven difficult because rhinos are elusive. Therefore, instead of tracking rhinos, we propose the novel approach of mapping communal defecation sites, called middens, which give information about rhinos' spatial behavior valuable to anti-poaching, management, and reintroduction efforts. This paper provides the first-ever mapping of rhino midden locations by building classifiers to detect them using remotely sensed thermal, RGB, and LiDAR imagery in passive and active learning settings. As existing active learning methods perform poorly due to the extreme class imbalance in our dataset, we design MultimodAL, an active learning system employing a ranking technique and multimodality to achieve competitive performance with passive learning models with 94% fewer labels. Our methods could therefore save over 76 hours in labeling time when used on a similarly-sized dataset. Unexpectedly, our midden map reveals that rhino middens are not randomly distributed throughout the landscape; rather, they are clustered. Consequently, rangers should be targeted at areas with high midden densities to strengthen anti-poaching efforts, in line with UN Target 15.7.

Title: E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding

• Authors: Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, Chang Wen Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18111
• Pdf URL: https://arxiv.org/pdf/2409.18111
• Copy Paste: [[2409.18111]] E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding(https://arxiv.org/abs/2409.18111)
• Keywords: large language model
• Abstract: Recent advances in Video Large Language Models (Video-LLMs) have demonstrated their great potential in general-purpose video understanding. To verify the significance of these models, a number of benchmarks have been proposed to diagnose their capabilities in different scenarios. However, existing benchmarks merely evaluate models through video-level question-answering, lacking fine-grained event-level assessment and task diversity. To fill this gap, we introduce E.T. Bench (Event-Level & Time-Sensitive Video Understanding Benchmark), a large-scale and high-quality benchmark for open-ended event-level video understanding. Categorized within a 3-level task taxonomy, E.T. Bench encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length) under 8 domains, providing comprehensive evaluations. We extensively evaluated 8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that state-of-the-art models for coarse-level (video-level) understanding struggle to solve our fine-grained tasks, e.g., grounding event-of-interests within videos, largely due to the short video context length, improper time representations, and lack of multi-event training data. Focusing on these issues, we further propose a strong baseline model, E.T. Chat, together with an instruction-tuning dataset E.T. Instruct 164K tailored for fine-grained event-level understanding. Our simple but effective solution demonstrates superior performance in multiple scenarios.

Title: EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation

• Authors: Jiaxiang Tang, Zhaoshuo Li, Zekun Hao, Xian Liu, Gang Zeng, Ming-Yu Liu, Qinsheng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18114
• Pdf URL: https://arxiv.org/pdf/2409.18114
• Copy Paste: [[2409.18114]] EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation(https://arxiv.org/abs/2409.18114)
• Keywords: diffusion
• Abstract: Current auto-regressive mesh generation methods suffer from issues such as incompleteness, insufficient detail, and poor generalization. In this paper, we propose an Auto-regressive Auto-encoder (ArAE) model capable of generating high-quality 3D meshes with up to 4,000 faces at a spatial resolution of $512^3$. We introduce a novel mesh tokenization algorithm that efficiently compresses triangular meshes into 1D token sequences, significantly enhancing training efficiency. Furthermore, our model compresses variable-length triangular meshes into a fixed-length latent space, enabling training latent diffusion models for better generalization. Extensive experiments demonstrate the superior quality, diversity, and generalization capabilities of our model in both point cloud and image-conditioned mesh generation tasks.

Title: Slowly Scaling Per-Record Differential Privacy

• Authors: Brian Finley, Anthony M Caruso, Justin C Doty, Ashwin Machanavajjhala, Mikaela R Meyer, David Pujol, William Sexton, Zachary Terner
• Subjects: cs.CR, stat.ME
• Abstract URL: https://arxiv.org/abs/2409.18118
• Pdf URL: https://arxiv.org/pdf/2409.18118
• Copy Paste: [[2409.18118]] Slowly Scaling Per-Record Differential Privacy(https://arxiv.org/abs/2409.18118)
• Keywords: privacy, protect, attack
• Abstract: We develop formal privacy mechanisms for releasing statistics from data with many outlying values, such as income data. These mechanisms ensure that a per-record differential privacy guarantee degrades slowly in the protected records' influence on the statistics being released. Formal privacy mechanisms generally add randomness, or "noise," to published statistics. If a noisy statistic's distribution changes little with the addition or deletion of a single record in the underlying dataset, an attacker looking at this statistic will find it plausible that any particular record was present or absent, preserving the records' privacy. More influential records -- those whose addition or deletion would change the statistics' distribution more -- typically suffer greater privacy loss. The per-record differential privacy framework quantifies these record-specific privacy guarantees, but existing mechanisms let these guarantees degrade rapidly (linearly or quadratically) with influence. While this may be acceptable in cases with some moderately influential records, it results in unacceptably high privacy losses when records' influence varies widely, as is common in economic data. We develop mechanisms with privacy guarantees that instead degrade as slowly as logarithmically with influence. These mechanisms allow for the accurate, unbiased release of statistics, while providing meaningful protection for highly influential records. As an example, we consider the private release of sums of unbounded establishment data such as payroll, where our mechanisms extend meaningful privacy protection even to very large establishments. We evaluate these mechanisms empirically and demonstrate their utility.

Title: Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography

• Authors: Yuexi Du, John Onofrey, Nicha C. Dvornek
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18119
• Pdf URL: https://arxiv.org/pdf/2409.18119
• Copy Paste: [[2409.18119]] Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography(https://arxiv.org/abs/2409.18119)
• Keywords: large language model
• Abstract: Contrastive Language-Image Pre-training (CLIP) shows promise in medical image analysis but requires substantial data and computational resources. Due to these restrictions, existing CLIP applications in medical imaging focus mainly on modalities like chest X-rays that have abundant image-report data available, leaving many other important modalities under-explored. Here, we propose the first adaptation of the full CLIP model to mammography, which presents significant challenges due to labeled data scarcity, high-resolution images with small regions of interest, and data imbalance. We first develop a specialized supervision framework for mammography that leverages its multi-view nature. Furthermore, we design a symmetric local alignment module to better focus on detailed features in high-resolution images. Lastly, we incorporate a parameter-efficient fine-tuning approach for large language models pre-trained with medical knowledge to address data limitations. Our multi-view and multi-scale alignment (MaMA) method outperforms state-of-the-art baselines for three different tasks on two large real-world mammography datasets, EMBED and RSNA-Mammo, with only 52% model size compared with the largest baseline.

Title: Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction

• Authors: Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Liu, Bingbing Liu, Ying-Cong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18124
• Pdf URL: https://arxiv.org/pdf/2409.18124
• Copy Paste: [[2409.18124]] Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction(https://arxiv.org/abs/2409.18124)
• Keywords: diffusion
• Abstract: Leveraging the visual priors of pre-trained text-to-image diffusion models offers a promising solution to enhance zero-shot generalization in dense prediction tasks. However, existing methods often uncritically use the original diffusion formulation, which may not be optimal due to the fundamental differences between dense prediction and image generation. In this paper, we provide a systemic analysis of the diffusion formulation for the dense prediction, focusing on both quality and efficiency. And we find that the original parameterization type for image generation, which learns to predict noise, is harmful for dense prediction; the multi-step noising/denoising diffusion process is also unnecessary and challenging to optimize. Based on these insights, we introduce Lotus, a diffusion-based visual foundation model with a simple yet effective adaptation protocol for dense prediction. Specifically, Lotus is trained to directly predict annotations instead of noise, thereby avoiding harmful variance. We also reformulate the diffusion process into a single-step procedure, simplifying optimization and significantly boosting inference speed. Additionally, we introduce a novel tuning strategy called detail preserver, which achieves more accurate and fine-grained predictions. Without scaling up the training data or model capacity, Lotus achieves SoTA performance in zero-shot depth and normal estimation across various datasets. It also significantly enhances efficiency, being hundreds of times faster than most existing diffusion-based methods.

Title: EgoLM: Multi-Modal Language Model of Egocentric Motions

• Authors: Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, Lingni Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18127
• Pdf URL: https://arxiv.org/pdf/2409.18127
• Copy Paste: [[2409.18127]] EgoLM: Multi-Modal Language Model of Egocentric Motions(https://arxiv.org/abs/2409.18127)
• Keywords: large language model
• Abstract: As the prevalence of wearable devices, learning egocentric motions becomes essential to develop contextual AI. In this work, we present EgoLM, a versatile framework that tracks and understands egocentric motions from multi-modal inputs, e.g., egocentric videos and motion sensors. EgoLM exploits rich contexts for the disambiguation of egomotion tracking and understanding, which are ill-posed under single modality conditions. To facilitate the versatile and multi-modal framework, our key insight is to model the joint distribution of egocentric motions and natural languages using large language models (LLM). Multi-modal sensor inputs are encoded and projected to the joint latent space of language models, and used to prompt motion generation or text generation for egomotion tracking or understanding, respectively. Extensive experiments on large-scale multi-modal human motion dataset validate the effectiveness of EgoLM as a generalist model for universal egocentric learning.

Title: FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner

• Authors: Wenliang Zhao, Minglei Shi, Xumin Yu, Jie Zhou, Jiwen Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18128
• Pdf URL: https://arxiv.org/pdf/2409.18128
• Copy Paste: [[2409.18128]] FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner(https://arxiv.org/abs/2409.18128)
• Keywords: diffusion, generative
• Abstract: Building on the success of diffusion models in visual generation, flow-based models reemerge as another prominent family of generative models that have achieved competitive or better performance in terms of both visual quality and inference speed. By learning the velocity field through flow-matching, flow-based models tend to produce a straighter sampling trajectory, which is advantageous during the sampling process. However, unlike diffusion models for which fast samplers are well-developed, efficient sampling of flow-based generative models has been rarely explored. In this paper, we propose a framework called FlowTurbo to accelerate the sampling of flow-based models while still enhancing the sampling quality. Our primary observation is that the velocity predictor's outputs in the flow-based models will become stable during the sampling, enabling the estimation of velocity via a lightweight velocity refiner. Additionally, we introduce several techniques including a pseudo corrector and sample-aware compilation to further reduce inference time. Since FlowTurbo does not change the multi-step sampling paradigm, it can be effectively applied for various tasks such as image editing, inpainting, etc. By integrating FlowTurbo into different flow-based models, we obtain an acceleration ratio of 53.1%$\sim$58.3% on class-conditional generation and 29.8%$\sim$38.5% on text-to-image generation. Notably, FlowTurbo reaches an FID of 2.12 on ImageNet with 100 (ms / img) and FID of 3.93 with 38 (ms / img), achieving the real-time image generation and establishing the new state-of-the-art. Code is available at this https URL.