2024-03-11

Title: QASE Enhanced PLMs: Improved Control in Text Generation for MRC

• Authors: Lin Ai, Zheng Hui, Zizhou Liu, Julia Hirschberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04771
• Pdf URL: https://arxiv.org/pdf/2403.04771
• Copy Paste: [[2403.04771]] QASE Enhanced PLMs: Improved Control in Text Generation for MRC(https://arxiv.org/abs/2403.04771)
• Keywords: extraction, generative
• Abstract: To address the challenges of out-of-control generation in generative models for machine reading comprehension (MRC), we introduce the Question-Attended Span Extraction (QASE) module. Integrated during the fine-tuning of pre-trained generative language models (PLMs), QASE enables these PLMs to match SOTA extractive methods and outperform leading LLMs like GPT-4 in MRC tasks, without significant increases in computational costs.

Title: An Efficient Difference-of-Convex Solver for Privacy Funnel

• Authors: Teng-Hui Huang, Hesham El Gamal
• Subjects: cs.LG, cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2403.04778
• Pdf URL: https://arxiv.org/pdf/2403.04778
• Copy Paste: [[2403.04778]] An Efficient Difference-of-Convex Solver for Privacy Funnel(https://arxiv.org/abs/2403.04778)
• Keywords: privacy
• Abstract: We propose an efficient solver for the privacy funnel (PF) method, leveraging its difference-of-convex (DC) structure. The proposed DC separation results in a closed-form update equation, which allows straightforward application to both known and unknown distribution settings. For known distribution case, we prove the convergence (local stationary points) of the proposed non-greedy solver, and empirically show that it outperforms the state-of-the-art approaches in characterizing the privacy-utility trade-off. The insights of our DC approach apply to unknown distribution settings where labeled empirical samples are available instead. Leveraging the insights, our alternating minimization solver satisfies the fundamental Markov relation of PF in contrast to previous variational inference-based solvers. Empirically, we evaluate the proposed solver with MNIST and Fashion-MNIST datasets. Our results show that under a comparable reconstruction quality, an adversary suffers from higher prediction error from clustering our compressed codes than that with the compared methods. Most importantly, our solver is independent to private information in inference phase contrary to the baselines.

Title: MuseGraph: Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining

• Authors: Yanchao Tan, Hang Lv, Xinyi Huang, Jiawei Zhang, Shiping Wang, Carl Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04780
• Pdf URL: https://arxiv.org/pdf/2403.04780
• Copy Paste: [[2403.04780]] MuseGraph: Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining(https://arxiv.org/abs/2403.04780)
• Keywords: generative, large language model
• Abstract: Graphs with abundant attributes are essential in modeling interconnected entities and improving predictions in various real-world applications. Traditional Graph Neural Networks (GNNs), which are commonly used for modeling attributed graphs, need to be re-trained every time when applied to different graph tasks and datasets. Although the emergence of Large Language Models (LLMs) has introduced a new paradigm in natural language processing, the generative potential of LLMs in graph mining remains largely under-explored. To this end, we propose a novel framework MuseGraph, which seamlessly integrates the strengths of GNNs and LLMs and facilitates a more effective and generic approach for graph mining across different tasks and datasets. Specifically, we first introduce a compact graph description via the proposed adaptive input generation to encapsulate key information from the graph under the constraints of language token limitations. Then, we propose a diverse instruction generation mechanism, which distills the reasoning capabilities from LLMs (e.g., GPT-4) to create task-specific Chain-of-Thought-based instruction packages for different graph tasks. Finally, we propose a graph-aware instruction tuning with a dynamic instruction package allocation strategy across tasks and datasets, ensuring the effectiveness and generalization of the training process. Our experimental results demonstrate significant improvements in different graph tasks, showcasing the potential of our MuseGraph in enhancing the accuracy of graph-oriented downstream tasks while keeping the generation powers of LLMs.

Title: Selective Encryption using Segmentation Mask with Chaotic Henon Map for Multidimensional Medical Images

• Authors: S Arut Prakash, Aditya Ganesh Kumar, Prabhu Shankar K. C., Lithicka Anandavel, Aditya Lakshmi Narayanan
• Subjects: cs.CR, cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.04781
• Pdf URL: https://arxiv.org/pdf/2403.04781
• Copy Paste: [[2403.04781]] Selective Encryption using Segmentation Mask with Chaotic Henon Map for Multidimensional Medical Images(https://arxiv.org/abs/2403.04781)
• Keywords: security, segmentation
• Abstract: A user-centric design and resource optimization should be at the center of any technology or innovation. The user-centric perspective gives the developer the opportunity to develop with task-based optimization. The user in the medical image field is a medical professional who analyzes the medical images and gives their diagnosis results to the patient. This scheme, having the medical professional user's perspective, innovates in the area of Medical Image storage and security. The architecture is designed with three main segments, namely: Segmentation, Storage, and Retrieval. This architecture was designed owing to the fact that the number of retrieval operations done by medical professionals was toweringly higher when compared to the storage operations done for some handful number of times for a particular medical image. This gives room for our innovation to segment out the medically indispensable part of the medical image, encrypt it, and store it. By encrypting the vital parts of the image using a strong encryption algorithm like the chaotic Henon map, we are able to keep the security intact. Now retrieving the medical image demands only the computationally less stressing decryption of the segmented region of interest. The decryption of the segmented region of interest results in the full recovery of the medical image which can be viewed on demand by the medical professionals for various diagnosis purposes. In this scheme, we were able to achieve a retrieval speed improvement of around 47% when compared to a full image encryption of brain medical CT images.

Title: AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks

• Authors: Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, Qingyun Wu
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.04783
• Pdf URL: https://arxiv.org/pdf/2403.04783
• Copy Paste: [[2403.04783]] AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks(https://arxiv.org/abs/2403.04783)
• Keywords: defense, attack, robust, large language model
• Abstract: Despite extensive pre-training and fine-tuning in moral alignment to prevent generating harmful information at user request, large language models (LLMs) remain vulnerable to jailbreak attacks. In this paper, we propose AutoDefense, a response-filtering based multi-agent defense framework that filters harmful responses from LLMs. This framework assigns different roles to LLM agents and employs them to complete the defense task collaboratively. The division in tasks enhances the overall instruction-following of LLMs and enables the integration of other defense components as tools. AutoDefense can adapt to various sizes and kinds of open-source LLMs that serve as agents. Through conducting extensive experiments on a large scale of harmful and safe prompts, we validate the effectiveness of the proposed AutoDefense in improving the robustness against jailbreak attacks, while maintaining the performance at normal user request. Our code and data are publicly available at https://github.com/XHMY/AutoDefense.

Title: Analysis of Privacy Leakage in Federated Large Language Models

• Authors: Minh N. Vu, Truc Nguyen, Tre' R. Jeter, My T. Thai
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04784
• Pdf URL: https://arxiv.org/pdf/2403.04784
• Copy Paste: [[2403.04784]] Analysis of Privacy Leakage in Federated Large Language Models(https://arxiv.org/abs/2403.04784)
• Keywords: privacy, protect, attack, membership infer, federate, large language model
• Abstract: With the rapid adoption of Federated Learning (FL) as the training and tuning protocol for applications utilizing Large Language Models (LLMs), recent research highlights the need for significant modifications to FL to accommodate the large-scale of LLMs. While substantial adjustments to the protocol have been introduced as a response, comprehensive privacy analysis for the adapted FL protocol is currently lacking. To address this gap, our work delves into an extensive examination of the privacy analysis of FL when used for training LLMs, both from theoretical and practical perspectives. In particular, we design two active membership inference attacks with guaranteed theoretical success rates to assess the privacy leakages of various adapted FL configurations. Our theoretical findings are translated into practical attacks, revealing substantial privacy vulnerabilities in popular LLMs, including BERT, RoBERTa, DistilBERT, and OpenAI's GPTs, across multiple real-world language datasets. Additionally, we conduct thorough experiments to evaluate the privacy leakage of these models when data is protected by state-of-the-art differential privacy (DP) mechanisms.

Title: Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data

• Authors: Jun-En Ding, Phan Nguyen Minh Thao, Wen-Chih Peng, Jian-Zhe Wang, Chun-Cheng Chug, Min-Chen Hsieh, Yun-Chien Tseng, Ling Chen, Dongsheng Luo, Chi-Te Wang, Pei-fu Chen, Feng Liu, Fang-Ming Hung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04785
• Pdf URL: https://arxiv.org/pdf/2403.04785
• Copy Paste: [[2403.04785]] Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data(https://arxiv.org/abs/2403.04785)
• Keywords: large language model
• Abstract: Chronic diseases such as diabetes are the leading causes of morbidity and mortality worldwide. Numerous research studies have been attempted with various deep learning models in diagnosis. However, most previous studies had certain limitations, including using publicly available datasets (e.g. MIMIC), and imbalanced data. In this study, we collected five-year electronic health records (EHRs) from the Taiwan hospital database, including 1,420,596 clinical notes, 387,392 laboratory test results, and more than 1,505 laboratory test items, focusing on research pre-training large language models. We proposed a novel Large Language Multimodal Models (LLMMs) framework incorporating multimodal data from clinical notes and laboratory test results for the prediction of chronic disease risk. Our method combined a text embedding encoder and multi-head attention layer to learn laboratory test values, utilizing a deep neural network (DNN) module to merge blood features with chronic disease semantics into a latent space. In our experiments, we observe that clinicalBERT and PubMed-BERT, when combined with attention fusion, can achieve an accuracy of 73% in multiclass chronic diseases and diabetes prediction. By transforming laboratory test values into textual descriptions and employing the Flan T-5 model, we achieved a 76% Area Under the ROC Curve (AUROC), demonstrating the effectiveness of leveraging numerical text data for training and inference in language models. This approach significantly improves the accuracy of early-stage diabetes prediction.

Title: Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models

• Authors: Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vaibhav Kumar, Vinija Jain, Aman Chadha
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04786
• Pdf URL: https://arxiv.org/pdf/2403.04786
• Copy Paste: [[2403.04786]] Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models(https://arxiv.org/abs/2403.04786)
• Keywords: security, privacy, defense, attack, robust, large language model
• Abstract: Large Language Models (LLMs) have become a cornerstone in the field of Natural Language Processing (NLP), offering transformative capabilities in understanding and generating human-like text. However, with their rising prominence, the security and vulnerability aspects of these models have garnered significant attention. This paper presents a comprehensive survey of the various forms of attacks targeting LLMs, discussing the nature and mechanisms of these attacks, their potential impacts, and current defense strategies. We delve into topics such as adversarial attacks that aim to manipulate model outputs, data poisoning that affects model training, and privacy concerns related to training data exploitation. The paper also explores the effectiveness of different attack methodologies, the resilience of LLMs against these attacks, and the implications for model integrity and user trust. By examining the latest research, we provide insights into the current landscape of LLM vulnerabilities and defense mechanisms. Our objective is to offer a nuanced understanding of LLM attacks, foster awareness within the AI community, and inspire robust solutions to mitigate these risks in future developments.

Title: TopicDiff: A Topic-enriched Diffusion Approach for Multimodal Conversational Emotion Detection

• Authors: Jiamin Luo, Jingjing Wang, Guodong Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04789
• Pdf URL: https://arxiv.org/pdf/2403.04789
• Copy Paste: [[2403.04789]] TopicDiff: A Topic-enriched Diffusion Approach for Multimodal Conversational Emotion Detection(https://arxiv.org/abs/2403.04789)
• Keywords: robust, diffusion
• Abstract: Multimodal Conversational Emotion (MCE) detection, generally spanning across the acoustic, vision and language modalities, has attracted increasing interest in the multimedia community. Previous studies predominantly focus on learning contextual information in conversations with only a few considering the topic information in single language modality, while always neglecting the acoustic and vision topic information. On this basis, we propose a model-agnostic Topic-enriched Diffusion (TopicDiff) approach for capturing multimodal topic information in MCE tasks. Particularly, we integrate the diffusion model into neural topic model to alleviate the diversity deficiency problem of neural topic model in capturing topic information. Detailed evaluations demonstrate the significant improvements of TopicDiff over the state-of-the-art MCE baselines, justifying the importance of multimodal topic information to MCE and the effectiveness of TopicDiff in capturing such information. Furthermore, we observe an interesting finding that the topic information in acoustic and vision is more discriminative and robust compared to the language.

Title: Online Training of Large Language Models: Learn while chatting

• Authors: Juhao Liang, Ziwei Wang, Zhuoheng Ma, Jianquan Li, Zhiyi Zhang, Xiangbo Wu, Benyou Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04790
• Pdf URL: https://arxiv.org/pdf/2403.04790
• Copy Paste: [[2403.04790]] Online Training of Large Language Models: Learn while chatting(https://arxiv.org/abs/2403.04790)
• Keywords: large language model
• Abstract: Large Language Models(LLMs) have dramatically revolutionized the field of Natural Language Processing(NLP), offering remarkable capabilities that have garnered widespread usage. However, existing interaction paradigms between LLMs and users are constrained by either inflexibility, limitations in customization, or a lack of persistent learning. This inflexibility is particularly evident as users, especially those without programming skills, have restricted avenues to enhance or personalize the model. Existing frameworks further complicate the model training and deployment process due to their computational inefficiencies and lack of user-friendly interfaces. To overcome these challenges, this paper introduces a novel interaction paradigm-'Online Training using External Interactions'-that merges the benefits of persistent, real-time model updates with the flexibility for individual customization through external interactions such as AI agents or online/offline knowledge bases.

Title: LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset

• Authors: Ahmed Izzidien, Holli Sargeant, Felix Steffek
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04791
• Pdf URL: https://arxiv.org/pdf/2403.04791
• Copy Paste: [[2403.04791]] LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset(https://arxiv.org/abs/2403.04791)
• Keywords: large language model
• Abstract: To undertake computational research of the law, efficiently identifying datasets of court decisions that relate to a specific legal issue is a crucial yet challenging endeavour. This study addresses the gap in the literature working with large legal corpora about how to isolate cases, in our case summary judgments, from a large corpus of UK court decisions. We introduce a comparative analysis of two computational methods: (1) a traditional natural language processing-based approach leveraging expert-generated keywords and logical operators and (2) an innovative application of the Claude 2 large language model to classify cases based on content-specific prompts. We use the Cambridge Law Corpus of 356,011 UK court decisions and determine that the large language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords. Despite iterative refinement, the search logic based on keywords fails to capture nuances in legal language. We identify and extract 3,102 summary judgment cases, enabling us to map their distribution across various UK courts over a temporal span. The paper marks a pioneering step in employing advanced natural language processing to tackle core legal research tasks, demonstrating how these technologies can bridge systemic gaps and enhance the accessibility of legal information. We share the extracted dataset metrics to support further research on summary judgments.

Title: Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?

• Authors: Yotam Intrator, Matan Halfon, Roman Goldenberg, Reut Tsarfaty, Matan Eyal, Ehud Rivlin, Yossi Matias, Natalia Aizenberg
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04792
• Pdf URL: https://arxiv.org/pdf/2403.04792
• Copy Paste: [[2403.04792]] Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?(https://arxiv.org/abs/2403.04792)
• Keywords: generative, large language model
• Abstract: Large language models hold significant promise in multilingual applications. However, inherent biases stemming from predominantly English-centric pre-training have led to the widespread practice of pre-translation, i.e., translating non-English inputs to English before inference, leading to complexity and information loss. This study re-evaluates the need for pre-translation in the context of PaLM2 models (Anil et al., 2023), which have been established as highly performant in multilingual tasks. We offer a comprehensive investigation across 108 languages and 6 diverse benchmarks, including open-end generative tasks, which were excluded from previous similar studies. Our findings challenge the pre-translation paradigm established in prior research, highlighting the advantages of direct inference in PaLM2. Specifically, PaLM2-L consistently outperforms pre-translation in 94 out of 108 languages. These findings pave the way for more efficient and effective multilingual applications, alleviating the limitations associated with pre-translation and unlocking linguistic authenticity.

Title: A Data-Driven Two-Phase Multi-Split Causal Ensemble Model for Time Series

• Authors: Zhipeng Ma, Marco Kemmerling, Daniel Buschmann, Chrismarie Enslin, Daniel Lütticke, Robert H. Schmitt
• Subjects: cs.LG, cs.AI, stat.ME
• Abstract URL: https://arxiv.org/abs/2403.04793
• Pdf URL: https://arxiv.org/pdf/2403.04793
• Copy Paste: [[2403.04793]] A Data-Driven Two-Phase Multi-Split Causal Ensemble Model for Time Series(https://arxiv.org/abs/2403.04793)
• Keywords: robust
• Abstract: Causal inference is a fundamental research topic for discovering the cause-effect relationships in many disciplines. However, not all algorithms are equally well-suited for a given dataset. For instance, some approaches may only be able to identify linear relationships, while others are applicable for non-linearities. Algorithms further vary in their sensitivity to noise and their ability to infer causal information from coupled vs. non-coupled time series. Therefore, different algorithms often generate different causal relationships for the same input. To achieve a more robust causal inference result, this publication proposes a novel data-driven two-phase multi-split causal ensemble model to combine the strengths of different causality base algorithms. In comparison to existing approaches, the proposed ensemble method reduces the influence of noise through a data partitioning scheme in the first phase. To achieve this, the data are initially divided into several partitions and the base algorithms are applied to each partition. Subsequently, Gaussian mixture models are used to identify the causal relationships derived from the different partitions that are likely to be valid. In the second phase, the identified relationships from each base algorithm are then merged based on three combination rules. The proposed ensemble approach is evaluated using multiple metrics, among them a newly developed evaluation index for causal ensemble approaches. We perform experiments using three synthetic datasets with different volumes and complexity, which are specifically designed to test causality detection methods under different circumstances while knowing the ground truth causal relationships. In these experiments, our causality ensemble outperforms each of its base algorithms. In practical applications, the use of the proposed method could hence lead to more robust and reliable causality results.

Title: Cloud Security Assurance: Strategies for Encryption in Digital Forensic Readiness

• Authors: Ahmed MohanRaj Alenezi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.04794
• Pdf URL: https://arxiv.org/pdf/2403.04794
• Copy Paste: [[2403.04794]] Cloud Security Assurance: Strategies for Encryption in Digital Forensic Readiness(https://arxiv.org/abs/2403.04794)
• Keywords: security, privacy, protect
• Abstract: This paper explores strategies for enhancing cloud security through encryption and digital forensic readiness. The adoption of cloud computing has brought unprecedented benefits to organizations but also introduces new security challenges. Encryption plays a crucial role in protecting data confidentiality and integrity within cloud environments. Various encryption techniques and key management practices are discussed, along with their implications for data privacy and regulatory compliance. Additionally, the paper examines the importance of digital forensic readiness in facilitating effective incident response and investigation in the cloud. Challenges associated with conducting digital forensics in cloud environments are addressed, and strategies for overcoming these challenges are proposed. By integrating encryption and digital forensic readiness into a cohesive security strategy, organizations can strengthen their resilience against emerging threats and maintain trust in their cloud-based operations.

Title: Large Language Models in Fire Engineering: An Examination of Technical Questions Against Domain Knowledge

• Authors: Haley Hostetter, M.Z. Naser, Xinyan Huang, John Gales
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04795
• Pdf URL: https://arxiv.org/pdf/2403.04795
• Copy Paste: [[2403.04795]] Large Language Models in Fire Engineering: An Examination of Technical Questions Against Domain Knowledge(https://arxiv.org/abs/2403.04795)
• Keywords: protect, large language model
• Abstract: This communication presents preliminary findings from comparing two recent chatbots, OpenAI's ChatGPT and Google's Bard, in the context of fire engineering by evaluating their responses in handling fire safety related queries. A diverse range of fire engineering questions and scenarios were created and examined, including structural fire design, fire prevention strategies, evacuation, building code compliance, and fire suppression systems (some of which resemble those commonly present in the Fire Protection exam (FPE)). The results reveal some key differences in the performance of the chatbots, with ChatGPT demonstrating a relatively superior performance. Then, this communication highlights the potential for chatbot technology to revolutionize fire engineering practices by providing instant access to critical information while outlining areas for further improvement and research. Evidently, and when it matures, this technology will likely be elemental to our engineers' practice and education.

Title: Blockchain-Enhanced UAV Networks for Post-Disaster Communication: A Decentralized Flocking Approach

• Authors: Sana Hafeez, Runze Cheng, Lina Mohjazi, Yao Sun, Muhammad Ali Imran
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.04796
• Pdf URL: https://arxiv.org/pdf/2403.04796
• Copy Paste: [[2403.04796]] Blockchain-Enhanced UAV Networks for Post-Disaster Communication: A Decentralized Flocking Approach(https://arxiv.org/abs/2403.04796)
• Keywords: secure, security, privacy, attack, robust
• Abstract: Unmanned Aerial Vehicles (UAVs) have significant potential for agile communication and relief coordination in post-disaster scenarios, particularly when ground infrastructure is compromised. However, efficiently coordinating and securing flocks of heterogeneous UAVs from different service providers poses significant challenges related to privacy, scalability, lightweight consensus protocols, and comprehensive cybersecurity mechanisms. This study introduces a robust blockchain-enabled framework designed to tackle these technical challenges through a combination of consensus protocols, smart contracts, and cryptographic techniques. First, we propose a consortium blockchain architecture that ensures secure and private multi-agency coordination by controlling access and safeguarding the privacy of sensitive data. Second, we develop an optimized hybrid consensus protocol that merges Delegated Proof of Stake and Practical Byzantine Fault Tolerance (DPOS-PBFT), aiming to achieve an effective balance between efficiency, security, and resilience against node failures. Finally, we introduce decentralized flocking algorithms that facilitate adaptable and autonomous operations among specialized UAV clusters, ensuring critical disaster relief functions under conditions of uncertain connectivity. Comprehensive simulations demonstrate the system achieved linear scaling of throughput up to 500 UAV nodes, with only a 50ms increase in latency from 10 to 500 nodes. The framework maintained high throughput and low latency despite spoofing, denial-of-service (DoS), and tampering attacks, showing strong cyber resilience. Communication latencies were kept under 10ms for diverse UAV operations through self-optimizing network intelligence, with median values around 2-3ms.

Title: Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

• Authors: Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04797
• Pdf URL: https://arxiv.org/pdf/2403.04797
• Copy Paste: [[2403.04797]] Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding(https://arxiv.org/abs/2403.04797)
• Keywords: large language model
• Abstract: This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are available at https://github.com/VITA-Group/Ms-PoE.

Title: Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs

• Authors: Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, Santu Rana
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04801
• Pdf URL: https://arxiv.org/pdf/2403.04801
• Copy Paste: [[2403.04801]] Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs(https://arxiv.org/abs/2403.04801)
• Keywords: attack
• Abstract: In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data directly, which is the dominant approach of quantifying memorization in LLMs. We use an iterative rejection-sampling optimization process to find instruction-based prompts with two main characteristics: (1) minimal overlap with the training data to avoid presenting the solution directly to the model, and (2) maximal overlap between the victim model's output and the training data, aiming to induce the victim to spit out training data. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that (1) instruction-tuned models can expose pre-training data as much as their base-models, if not more so, (2) contexts other than the original training data can lead to leakage, and (3) using instructions proposed by other LLMs can open a new avenue of automated attacks that we should further study and explore. The code can be found at https://github.com/Alymostafa/Instruction_based_attack .

Title: Enhancing Security in Federated Learning through Adaptive Consensus-Based Model Update Validation

• Authors: Zahir Alsulaimawi
• Subjects: cs.CR, cs.AI, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04803
• Pdf URL: https://arxiv.org/pdf/2403.04803
• Copy Paste: [[2403.04803]] Enhancing Security in Federated Learning through Adaptive Consensus-Based Model Update Validation(https://arxiv.org/abs/2403.04803)
• Keywords: secure, security, privacy, attack, robust, federate
• Abstract: This paper introduces an advanced approach for fortifying Federated Learning (FL) systems against label-flipping attacks. We propose a simplified consensus-based verification process integrated with an adaptive thresholding mechanism. This dynamic thresholding is designed to adjust based on the evolving landscape of model updates, offering a refined layer of anomaly detection that aligns with the real-time needs of distributed learning environments. Our method necessitates a majority consensus among participating clients to validate updates, ensuring that only vetted and consensual modifications are applied to the global model. The efficacy of our approach is validated through experiments on two benchmark datasets in deep learning, CIFAR-10 and MNIST. Our results indicate a significant mitigation of label-flipping attacks, bolstering the FL system's resilience. This method transcends conventional techniques that depend on anomaly detection or statistical validation by incorporating a verification layer reminiscent of blockchain's participatory validation without the associated cryptographic overhead. The innovation of our approach rests in striking an optimal balance between heightened security measures and the inherent limitations of FL systems, such as computational efficiency and data privacy. Implementing a consensus mechanism specifically tailored for FL environments paves the way for more secure, robust, and trustworthy distributed machine learning applications, where safeguarding data integrity and model robustness is critical.

Title: Not all tickets are equal and we know it: Guiding pruning with domain-specific knowledge

• Authors: Intekhab Hossain, Jonas Fischer, Rebekka Burkholz, John Quackenbush
• Subjects: cs.LG, q-bio.QM, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.04805
• Pdf URL: https://arxiv.org/pdf/2403.04805
• Copy Paste: [[2403.04805]] Not all tickets are equal and we know it: Guiding pruning with domain-specific knowledge(https://arxiv.org/abs/2403.04805)
• Keywords: interpretability
• Abstract: Neural structure learning is of paramount importance for scientific discovery and interpretability. Yet, contemporary pruning algorithms that focus on computational resource efficiency face algorithmic barriers to select a meaningful model that aligns with domain expertise. To mitigate this challenge, we propose DASH, which guides pruning by available domain-specific structural information. In the context of learning dynamic gene regulatory network models, we show that DASH combined with existing general knowledge on interaction partners provides data-specific insights aligned with biology. For this task, we show on synthetic data with ground truth information and two real world applications the effectiveness of DASH, which outperforms competing methods by a large margin and provides more meaningful biological insights. Our work shows that domain specific structural information bears the potential to improve model-derived scientific insights.

Title: WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off

• Authors: Eva Giboulot, Furon Teddy
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04808
• Pdf URL: https://arxiv.org/pdf/2403.04808
• Copy Paste: [[2403.04808]] WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off(https://arxiv.org/abs/2403.04808)
• Keywords: robust, watermark, large language model
• Abstract: Watermarking is a technical means to dissuade malfeasant usage of Large Language Models. This paper proposes a novel watermarking scheme, so-called WaterMax, that enjoys high detectability while sustaining the quality of the generated text of the original LLM. Its new design leaves the LLM untouched (no modification of the weights, logits, temperature, or sampling technique). WaterMax balances robustness and complexity contrary to the watermarking techniques of the literature inherently provoking a trade-off between quality and robustness. Its performance is both theoretically proven and experimentally validated. It outperforms all the SotA techniques under the most complete benchmark suite.

Title: Restricted Bayesian Neural Network

• Authors: Sourav Ganguly
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.04810
• Pdf URL: https://arxiv.org/pdf/2403.04810
• Copy Paste: [[2403.04810]] Restricted Bayesian Neural Network(https://arxiv.org/abs/2403.04810)
• Keywords: robust
• Abstract: Modern deep learning tools are remarkably effective in addressing intricate problems. However, their operation as black-box models introduces increased uncertainty in predictions. Additionally, they contend with various challenges, including the need for substantial storage space in large networks, issues of overfitting, underfitting, vanishing gradients, and more. This study explores the concept of Bayesian Neural Networks, presenting a novel architecture designed to significantly alleviate the storage space complexity of a network. Furthermore, we introduce an algorithm adept at efficiently handling uncertainties, ensuring robust convergence values without becoming trapped in local optima, particularly when the objective function lacks perfect convexity.

Title: TrafPS: A Shapley-based Visual Analytics Approach to Interpret Traffic

• Authors: Zezheng Feng, Yifan Jiang, Hongjun Wang, Zipei Fan, Yuxin Ma, Shuang-Hua Yang, Huamin Qu, Xuan Song
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.04812
• Pdf URL: https://arxiv.org/pdf/2403.04812
• Copy Paste: [[2403.04812]] TrafPS: A Shapley-based Visual Analytics Approach to Interpret Traffic(https://arxiv.org/abs/2403.04812)
• Keywords: interpretability
• Abstract: Recent achievements in deep learning (DL) have shown its potential for predicting traffic flows. Such predictions are beneficial for understanding the situation and making decisions in traffic control. However, most state-of-the-art DL models are considered "black boxes" with little to no transparency for end users with respect to the underlying mechanisms. Some previous work tried to "open the black boxes" and increase the interpretability of how predictions are generated. However, it still remains challenging to handle complex models on large-scale spatio-temporal data and discover salient spatial and temporal patterns that significantly influence traffic flows. To overcome the challenges, we present TrafPS, a visual analytics approach for interpreting traffic prediction outcomes to support decision-making in traffic management and urban planning. The measurements, region SHAP and trajectory SHAP, are proposed to quantify the impact of flow patterns on urban traffic at different levels. Based on the task requirement from the domain experts, we employ an interactive visual interface for multi-aspect exploration and analysis of significant flow patterns. Two real-world case studies demonstrate the effectiveness of TrafPS in identifying key routes and decision-making support for urban planning.

Title: Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks

• Authors: Linyuan Gong, Sida Wang, Mostafa Elhoushi, Alvin Cheung
• Subjects: cs.CL, cs.AI, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.04814
• Pdf URL: https://arxiv.org/pdf/2403.04814
• Copy Paste: [[2403.04814]] Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks(https://arxiv.org/abs/2403.04814)
• Keywords: robust, fair, large language model
• Abstract: We introduce Syntax-Aware Fill-In-the-Middle (SAFIM), a new benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. This benchmark focuses on syntax-aware completions of program structures such as code blocks and conditional expressions, and includes 17,720 examples from multiple programming languages, sourced from recent code submissions after April 2022 to minimize data contamination. SAFIM provides a robust framework with various prompt designs and novel syntax-aware post-processing techniques, facilitating accurate and fair comparisons across LLMs. Our comprehensive evaluation of 15 LLMs shows that FIM pretraining not only enhances FIM proficiency but also improves Left-to-Right (L2R) inference using LLMs. Our findings challenge conventional beliefs and suggest that pretraining methods and data quality have more impact than model size. SAFIM thus serves as a foundational platform for future research in effective pretraining strategies for code LLMs. The evaluation toolkit and dataset are available at https://github.com/gonglinyuan/safim, and the leaderboard is available at https://safimbenchmark.com.

Title: Automating the Information Extraction from Semi-Structured Interview Transcripts

• Authors: Angelina Parfenova
• Subjects: cs.CL, cs.CY, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.04819
• Pdf URL: https://arxiv.org/pdf/2403.04819
• Copy Paste: [[2403.04819]] Automating the Information Extraction from Semi-Structured Interview Transcripts(https://arxiv.org/abs/2403.04819)
• Keywords: extraction
• Abstract: This paper explores the development and application of an automated system designed to extract information from semi-structured interview transcripts. Given the labor-intensive nature of traditional qualitative analysis methods, such as coding, there exists a significant demand for tools that can facilitate the analysis process. Our research investigates various topic modeling techniques and concludes that the best model for analyzing interview texts is a combination of BERT embeddings and HDBSCAN clustering. We present a user-friendly software prototype that enables researchers, including those without programming skills, to efficiently process and visualize the thematic structure of interview data. This tool not only facilitates the initial stages of qualitative analysis but also offers insights into the interconnectedness of topics revealed, thereby enhancing the depth of qualitative analysis.

Title: Solving Inverse Problems with Model Mismatch using Untrained Neural Networks within Model-based Architectures

• Authors: Peimeng Guan, Naveed Iqbal, Mark A. Davenport, Mudassir Masood
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.04847
• Pdf URL: https://arxiv.org/pdf/2403.04847
• Copy Paste: [[2403.04847]] Solving Inverse Problems with Model Mismatch using Untrained Neural Networks within Model-based Architectures(https://arxiv.org/abs/2403.04847)
• Keywords: robust
• Abstract: Model-based deep learning methods such as \emph{loop unrolling} (LU) and \emph{deep equilibrium model} (DEQ) extensions offer outstanding performance in solving inverse problems (IP). These methods unroll the optimization iterations into a sequence of neural networks that in effect learn a regularization function from data. While these architectures are currently state-of-the-art in numerous applications, their success heavily relies on the accuracy of the forward model. This assumption can be limiting in many physical applications due to model simplifications or uncertainties in the apparatus. To address forward model mismatch, we introduce an untrained forward model residual block within the model-based architecture to match the data consistency in the measurement domain for each instance. We propose two variants in well-known model-based architectures (LU and DEQ) and prove convergence under mild conditions. The experiments show significant quality improvement in removing artifacts and preserving details across three distinct applications, encompassing both linear and nonlinear inverse problems. Moreover, we highlight reconstruction effectiveness in intermediate steps and showcase robustness to random initialization of the residual block and a higher number of iterations during evaluation.

Title: Evaluating Biases in Context-Dependent Health Questions

• Authors: Sharon Levy, Tahilin Sanchez Karver, William D. Adler, Michelle R. Kaufman, Mark Dredze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04858
• Pdf URL: https://arxiv.org/pdf/2403.04858
• Copy Paste: [[2403.04858]] Evaluating Biases in Context-Dependent Health Questions(https://arxiv.org/abs/2403.04858)
• Keywords: large language model
• Abstract: Chat-based large language models have the opportunity to empower individuals lacking high-quality healthcare access to receive personalized information across a variety of topics. However, users may ask underspecified questions that require additional context for a model to correctly answer. We study how large language model biases are exhibited through these contextual questions in the healthcare domain. To accomplish this, we curate a dataset of sexual and reproductive healthcare questions that are dependent on age, sex, and location attributes. We compare models' outputs with and without demographic context to determine group alignment among our contextual questions. Our experiments reveal biases in each of these attributes, where young adult female users are favored.

Title: Group Privacy Amplification and Unified Amplification by Subsampling for Rényi Differential Privacy

• Authors: Jan Schuchardt, Mihail Stoian, Arthur Kosmala, Stephan Günnemann
• Subjects: cs.CR, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.04867
• Pdf URL: https://arxiv.org/pdf/2403.04867
• Copy Paste: [[2403.04867]] Group Privacy Amplification and Unified Amplification by Subsampling for Rényi Differential Privacy(https://arxiv.org/abs/2403.04867)
• Keywords: privacy, robust
• Abstract: Differential privacy (DP) has various desirable properties, such as robustness to post-processing, group privacy, and amplification by subsampling, which can be derived independently of each other. Our goal is to determine whether stronger privacy guarantees can be obtained by considering multiple of these properties jointly. To this end, we focus on the combination of group privacy and amplification by subsampling. To provide guarantees that are amenable to machine learning algorithms, we conduct our analysis in the framework of R\'enyi-DP, which has more favorable composition properties than $(\epsilon,\delta)$-DP. As part of this analysis, we develop a unified framework for deriving amplification by subsampling guarantees for R\'enyi-DP, which represents the first such framework for a privacy accounting method and is of independent interest. We find that it not only lets us improve upon and generalize existing amplification results for R\'enyi-DP, but also derive provably tight group privacy amplification guarantees stronger than existing principles. These results establish the joint study of different DP properties as a promising research direction.

Title: An Item is Worth a Prompt: Versatile Image Editing with Disentangled Control

• Authors: Aosong Feng, Weikang Qiu, Jinbin Bai, Kaicheng Zhou, Zhen Dong, Xiao Zhang, Rex Ying, Leandros Tassiulas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04880
• Pdf URL: https://arxiv.org/pdf/2403.04880
• Copy Paste: [[2403.04880]] An Item is Worth a Prompt: Versatile Image Editing with Disentangled Control(https://arxiv.org/abs/2403.04880)
• Keywords: diffusion
• Abstract: Building on the success of text-to-image diffusion models (DPMs), image editing is an important application to enable human interaction with AI-generated content. Among various editing methods, editing within the prompt space gains more attention due to its capacity and simplicity of controlling semantics. However, since diffusion models are commonly pretrained on descriptive text captions, direct editing of words in text prompts usually leads to completely different generated images, violating the requirements for image editing. On the other hand, existing editing methods usually consider introducing spatial masks to preserve the identity of unedited regions, which are usually ignored by DPMs and therefore lead to inharmonic editing results. Targeting these two challenges, in this work, we propose to disentangle the comprehensive image-prompt interaction into several item-prompt interactions, with each item linked to a special learned prompt. The resulting framework, named D-Edit, is based on pretrained diffusion models with cross-attention layers disentangled and adopts a two-step optimization to build item-prompt associations. Versatile image editing can then be applied to specific items by manipulating the corresponding prompts. We demonstrate state-of-the-art results in four types of editing operations including image-based, text-based, mask-based editing, and item removal, covering most types of editing applications, all within a single unified framework. Notably, D-Edit is the first framework that can (1) achieve item editing through mask editing and (2) combine image and text-based editing. We demonstrate the quality and versatility of the editing results for a diverse collection of images through both qualitative and quantitative evaluations.

Title: Efficient High-Resolution Time Series Classification via Attention Kronecker Decomposition

• Authors: Aosong Feng, Jialin Chen, Juan Garza, Brooklyn Berry, Francisco Salazar, Yifeng Gao, Rex Ying, Leandros Tassiulas
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04882
• Pdf URL: https://arxiv.org/pdf/2403.04882
• Copy Paste: [[2403.04882]] Efficient High-Resolution Time Series Classification via Attention Kronecker Decomposition(https://arxiv.org/abs/2403.04882)
• Keywords: robust, transformer
• Abstract: The high-resolution time series classification problem is essential due to the increasing availability of detailed temporal data in various domains. To tackle this challenge effectively, it is imperative that the state-of-the-art attention model is scalable to accommodate the growing sequence lengths typically encountered in high-resolution time series data, while also demonstrating robustness in handling the inherent noise prevalent in such datasets. To address this, we propose to hierarchically encode the long time series into multiple levels based on the interaction ranges. By capturing relationships at different levels, we can build more robust, expressive, and efficient models that are capable of capturing both short-term fluctuations and long-term trends in the data. We then propose a new time series transformer backbone (KronTime) by introducing Kronecker-decomposed attention to process such multi-level time series, which sequentially calculates attention from the lower level to the upper level. Experiments on four long time series datasets demonstrate superior classification results with improved efficiency compared to baseline methods.

Title: Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering

• Authors: Ojas Gramopadhye, Saeel Sandeep Nachane, Prateek Chanda, Ganesh Ramakrishnan, Kshitij Sharad Jadhav, Yatin Nandwani, Dinesh Raghu, Sachindra Joshi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04890
• Pdf URL: https://arxiv.org/pdf/2403.04890
• Copy Paste: [[2403.04890]] Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering(https://arxiv.org/abs/2403.04890)
• Keywords: large language model
• Abstract: Large Language models (LLMs) have demonstrated significant potential in transforming healthcare by automating tasks such as clinical documentation, information retrieval, and decision support. In this aspect, carefully engineered prompts have emerged as a powerful tool for using LLMs for medical scenarios, e.g., patient clinical scenarios. In this paper, we propose a modified version of the MedQA-USMLE dataset, which is subjective, to mimic real-life clinical scenarios. We explore the Chain of Thought (CoT) reasoning based on subjective response generation for the modified MedQA-USMLE dataset with appropriate LM-driven forward reasoning for correct responses to the medical questions. Keeping in mind the importance of response verification in the medical setting, we utilize a reward training mechanism whereby the language model also provides an appropriate verified response for a particular response to a clinical question. In this regard, we also include human-in-the-loop for different evaluation aspects. We develop better in-contrast learning strategies by modifying the 5-shot-codex-CoT-prompt from arXiv:2207.08143 for the subjective MedQA dataset and developing our incremental-reasoning prompt. Our evaluations show that the incremental reasoning prompt performs better than the modified codex prompt in certain scenarios. We also show that greedy decoding with the incremental reasoning method performs better than other strategies, such as prompt chaining and eliminative reasoning.

Title: ConstitutionalExperts: Training a Mixture of Principle-based Prompts

• Authors: Savvas Petridis, Ben Wedin, Ann Yuan, James Wexler, Nithum Thain
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04894
• Pdf URL: https://arxiv.org/pdf/2403.04894
• Copy Paste: [[2403.04894]] ConstitutionalExperts: Training a Mixture of Principle-based Prompts(https://arxiv.org/abs/2403.04894)
• Keywords: large language model
• Abstract: Large language models (LLMs) are highly capable at a variety of tasks given the right prompt, but writing one is still a difficult and tedious process. In this work, we introduce ConstitutionalExperts, a method for learning a prompt consisting of constitutional principles (i.e. rules), given a training dataset. Unlike prior methods that optimize the prompt as a single entity, our method incrementally improves the prompt by surgically editing individual principles. We also show that we can improve overall performance by learning unique prompts for different semantic regions of the training data and using a mixture-of-experts (MoE) architecture to route inputs at inference time. We compare our method to other state of the art prompt-optimization techniques across six benchmark datasets. We also investigate whether MoE improves these other techniques. Our results suggest that ConstitutionalExperts outperforms other prompt optimization techniques by 10.9% (F1) and that mixture-of-experts improves all techniques, suggesting its broad applicability.

Title: Secure Information Embedding and Extraction in Forensic 3D Fingerprinting

• Authors: Canran Wang, Jinwen Wang, Mi Zhou, Vinh Pham, Senyue Hao, Chao Zhou, Ning Zhang, Netanel Raviv
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.04918
• Pdf URL: https://arxiv.org/pdf/2403.04918
• Copy Paste: [[2403.04918]] Secure Information Embedding and Extraction in Forensic 3D Fingerprinting(https://arxiv.org/abs/2403.04918)
• Keywords: secure, security, extraction
• Abstract: The prevalence of 3D printing poses a significant risk to public safety, as any individual with internet access and a commodity printer is able to produce untraceable firearms, keys, counterfeit products, etc. To aid government authorities in combating these new security threats, several approaches have been taken to tag 3D-prints with identifying information. Known as fingerprints, this information is written into the object using various bit embedding techniques; examples include varying the height of the molten thermoplastic layers, and depositing metallic powder with different magnetic properties. Yet, the practicality of theses techniques in real-world forensic settings is hindered by the adversarial nature of this problem. That is, the 3D-printing process is out of reach of any law enforcement agencies; it is the adversary who controls all aspects of printing and possesses the printed object. To combat these threats, law enforcement agencies can regulate the manufacturing of 3D printers, on which they may enforce a fingerprinting scheme, and collect adversarially tampered remains (e.g., fragments of a broken 3D-printed firearm) during forensic investigation. Therefore, it is important to devise fingerprinting techniques so that the fingerprint could be extracted even if printing is carried out by the adversary. To this end, we present SIDE (Secure Information Embedding and Extraction), a fingerprinting framework that tackles the adversarial nature of forensic fingerprinting in 3D prints by offering both secure information embedding and secure information extraction.

Title: $\text{R}^2$-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations

• Authors: Xiang Li, Kai Qiu, Jinglu Wang, Xiaohao Xu, Rita Singh, Kashu Yamazak, Hao Chen, Xiaonan Huang, Bhiksha Raj
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04924
• Pdf URL: https://arxiv.org/pdf/2403.04924
• Copy Paste: [[2403.04924]] $\text{R}^2$-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations(https://arxiv.org/abs/2403.04924)
• Keywords: robust
• Abstract: Referring perception, which aims at grounding visual objects with multimodal referring guidance, is essential for bridging the gap between humans, who provide instructions, and the environment where intelligent systems perceive. Despite progress in this field, the robustness of referring perception models (RPMs) against disruptive perturbations is not well explored. This work thoroughly assesses the resilience of RPMs against various perturbations in both general and specific contexts. Recognizing the complex nature of referring perception tasks, we present a comprehensive taxonomy of perturbations, and then develop a versatile toolbox for synthesizing and evaluating the effects of composite disturbances. Employing this toolbox, we construct $\text{R}^2$-Bench, a benchmark for assessing the Robustness of Referring perception models under noisy conditions across five key tasks. Moreover, we propose the $\text{R}^2$-Agent, an LLM-based agent that simplifies and automates model evaluation via natural language instructions. Our investigation uncovers the vulnerabilities of current RPMs to various perturbations and provides tools for assessing model robustness, potentially promoting the safe and resilient integration of intelligent systems into complex real-world scenarios.

Title: BAGS: Blur Agnostic Gaussian Splatting through Multi-Scale Kernel Modeling

• Authors: Cheng Peng, Yutao Tang, Yifan Zhou, Nengyu Wang, Xijun Liu, Deming Li, Rama Chellappa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04926
• Pdf URL: https://arxiv.org/pdf/2403.04926
• Copy Paste: [[2403.04926]] BAGS: Blur Agnostic Gaussian Splatting through Multi-Scale Kernel Modeling(https://arxiv.org/abs/2403.04926)
• Keywords: robust
• Abstract: Recent efforts in using 3D Gaussians for scene reconstruction and novel view synthesis can achieve impressive results on curated benchmarks; however, images captured in real life are often blurry. In this work, we analyze the robustness of Gaussian-Splatting-based methods against various image blur, such as motion blur, defocus blur, downscaling blur, \etc. Under these degradations, Gaussian-Splatting-based methods tend to overfit and produce worse results than Neural-Radiance-Field-based methods. To address this issue, we propose Blur Agnostic Gaussian Splatting (BAGS). BAGS introduces additional 2D modeling capacities such that a 3D-consistent and high quality scene can be reconstructed despite image-wise blur. Specifically, we model blur by estimating per-pixel convolution kernels from a Blur Proposal Network (BPN). BPN is designed to consider spatial, color, and depth variations of the scene to maximize modeling capacity. Additionally, BPN also proposes a quality-assessing mask, which indicates regions where blur occur. Finally, we introduce a coarse-to-fine kernel optimization scheme; this optimization scheme is fast and avoids sub-optimal solutions due to a sparse point cloud initialization, which often occurs when we apply Structure-from-Motion on blurry images. We demonstrate that BAGS achieves photorealistic renderings under various challenging blur conditions and imaging geometry, while significantly improving upon existing approaches.

Title: On the Markov Property of Neural Algorithmic Reasoning: Analyses and Methods

• Authors: Montgomery Bohde, Meng Liu, Alexandra Saxton, Shuiwang Ji
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.04929
• Pdf URL: https://arxiv.org/pdf/2403.04929
• Copy Paste: [[2403.04929]] On the Markov Property of Neural Algorithmic Reasoning: Analyses and Methods(https://arxiv.org/abs/2403.04929)
• Keywords: robust
• Abstract: Neural algorithmic reasoning is an emerging research direction that endows neural networks with the ability to mimic algorithmic executions step-by-step. A common paradigm in existing designs involves the use of historical embeddings in predicting the results of future execution steps. Our observation in this work is that such historical dependence intrinsically contradicts the Markov nature of algorithmic reasoning tasks. Based on this motivation, we present our ForgetNet, which does not use historical embeddings and thus is consistent with the Markov nature of the tasks. To address challenges in training ForgetNet at early stages, we further introduce G-ForgetNet, which uses a gating mechanism to allow for the selective integration of historical embeddings. Such an enhanced capability provides valuable computational pathways during the model's early training phase. Our extensive experiments, based on the CLRS-30 algorithmic reasoning benchmark, demonstrate that both ForgetNet and G-ForgetNet achieve better generalization capability than existing methods. Furthermore, we investigate the behavior of the gating mechanism, highlighting its degree of alignment with our intuitions and its effectiveness for robust performance.

Title: AFreeCA: Annotation-Free Counting for All

• Authors: Adriano D'Alessandro, Ali Mahdavi-Amiri, Ghassan Hamarneh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04943
• Pdf URL: https://arxiv.org/pdf/2403.04943
• Copy Paste: [[2403.04943]] AFreeCA: Annotation-Free Counting for All(https://arxiv.org/abs/2403.04943)
• Keywords: robust, diffusion
• Abstract: Object counting methods typically rely on manually annotated datasets. The cost of creating such datasets has restricted the versatility of these networks to count objects from specific classes (such as humans or penguins), and counting objects from diverse categories remains a challenge. The availability of robust text-to-image latent diffusion models (LDMs) raises the question of whether these models can be utilized to generate counting datasets. However, LDMs struggle to create images with an exact number of objects based solely on text prompts but they can be used to offer a dependable \textit{sorting} signal by adding and removing objects within an image. Leveraging this data, we initially introduce an unsupervised sorting methodology to learn object-related features that are subsequently refined and anchored for counting purposes using counting data generated by LDMs. Further, we present a density classifier-guided method for dividing an image into patches containing objects that can be reliably counted. Consequently, we can generate counting data for any type of object and count them in an unsupervised manner. Our approach outperforms other unsupervised and few-shot alternatives and is not restricted to specific object classes for which counting data is available. Code to be released upon acceptance.

Title: Fooling Neural Networks for Motion Forecasting via Adversarial Attacks

• Authors: Edgar Medina, Leyong Loh
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04954
• Pdf URL: https://arxiv.org/pdf/2403.04954
• Copy Paste: [[2403.04954]] Fooling Neural Networks for Motion Forecasting via Adversarial Attacks(https://arxiv.org/abs/2403.04954)
• Keywords: attack
• Abstract: Human motion prediction is still an open problem, which is extremely important for autonomous driving and safety applications. Although there are great advances in this area, the widely studied topic of adversarial attacks has not been applied to multi-regression models such as GCNs and MLP-based architectures in human motion prediction. This work intends to reduce this gap using extensive quantitative and qualitative experiments in state-of-the-art architectures similar to the initial stages of adversarial attacks in image classification. The results suggest that models are susceptible to attacks even on low levels of perturbation. We also show experiments with 3D transformations that affect the model performance, in particular, we show that most models are sensitive to simple rotations and translations which do not alter joint distances. We conclude that similar to earlier CNN models, motion forecasting tasks are susceptible to small perturbations and simple 3D transformations.

Title: SecGPT: An Execution Isolation Architecture for LLM-Based Systems

• Authors: Yuhao Wu, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, Umar Iqbal
• Subjects: cs.CR, cs.AI, cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04960
• Pdf URL: https://arxiv.org/pdf/2403.04960
• Copy Paste: [[2403.04960]] SecGPT: An Execution Isolation Architecture for LLM-Based Systems(https://arxiv.org/abs/2403.04960)
• Keywords: security, privacy, protect, attack, large language model
• Abstract: Large language models (LLMs) extended as systems, such as ChatGPT, have begun supporting third-party applications. These LLM apps leverage the de facto natural language-based automated execution paradigm of LLMs: that is, apps and their interactions are defined in natural language, provided access to user data, and allowed to freely interact with each other and the system. These LLM app ecosystems resemble the settings of earlier computing platforms, where there was insufficient isolation between apps and the system. Because third-party apps may not be trustworthy, and exacerbated by the imprecision of the natural language interfaces, the current designs pose security and privacy risks for users. In this paper, we propose SecGPT, an architecture for LLM-based systems that aims to mitigate the security and privacy issues that arise with the execution of third-party apps. SecGPT's key idea is to isolate the execution of apps and more precisely mediate their interactions outside of their isolated environments. We evaluate SecGPT against a number of case study attacks and demonstrate that it protects against many security, privacy, and safety issues that exist in non-isolated LLM-based systems. The performance overhead incurred by SecGPT to improve security is under 0.3x for three-quarters of the tested queries. To foster follow-up research, we release SecGPT's source code at https://github.com/llm-platform-security/SecGPT.

Title: An In-depth Evaluation of GPT-4 in Sentence Simplification with Error-based Human Assessment

• Authors: Xuanxin Wu, Yuki Arase
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04963
• Pdf URL: https://arxiv.org/pdf/2403.04963
• Copy Paste: [[2403.04963]] An In-depth Evaluation of GPT-4 in Sentence Simplification with Error-based Human Assessment(https://arxiv.org/abs/2403.04963)
• Keywords: large language model
• Abstract: Sentence simplification, which rewrites a sentence to be easier to read and understand, is a promising technique to help people with various reading difficulties. With the rise of advanced large language models (LLMs), evaluating their performance in sentence simplification has become imperative. Recent studies have used both automatic metrics and human evaluations to assess the simplification abilities of LLMs. However, the suitability of existing evaluation methodologies for LLMs remains in question. First, the suitability of current automatic metrics on LLMs' simplification evaluation is still uncertain. Second, current human evaluation approaches in sentence simplification often fall into two extremes: they are either too superficial, failing to offer a clear understanding of the models' performance, or overly detailed, making the annotation process complex and prone to inconsistency, which in turn affects the evaluation's reliability. To address these problems, this study provides in-depth insights into LLMs' performance while ensuring the reliability of the evaluation. We design an error-based human annotation framework to assess the GPT-4's simplification capabilities. Results show that GPT-4 generally generates fewer erroneous simplification outputs compared to the current state-of-the-art. However, LLMs have their limitations, as seen in GPT-4's struggles with lexical paraphrasing. Furthermore, we conduct meta-evaluations on widely used automatic metrics using our human annotations. We find that while these metrics are effective for significant quality differences, they lack sufficient sensitivity to assess the overall high-quality simplification by GPT-4.

Title: StereoDiffusion: Training-Free Stereo Image Generation Using Latent Diffusion Models

• Authors: Lezhong Wang, Jeppe Revall Frisvad, Mark Bo Jensen, Siavash Arjomand Bigdeli
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04965
• Pdf URL: https://arxiv.org/pdf/2403.04965
• Copy Paste: [[2403.04965]] StereoDiffusion: Training-Free Stereo Image Generation Using Latent Diffusion Models(https://arxiv.org/abs/2403.04965)
• Keywords: diffusion
• Abstract: The demand for stereo images increases as manufacturers launch more XR devices. To meet this demand, we introduce StereoDiffusion, a method that, unlike traditional inpainting pipelines, is trainning free, remarkably straightforward to use, and it seamlessly integrates into the original Stable Diffusion model. Our method modifies the latent variable to provide an end-to-end, lightweight capability for fast generation of stereo image pairs, without the need for fine-tuning model weights or any post-processing of images. Using the original input to generate a left image and estimate a disparity map for it, we generate the latent vector for the right image through Stereo Pixel Shift operations, complemented by Symmetric Pixel Shift Masking Denoise and Self-Attention Layers Modification methods to align the right-side image with the left-side image. Moreover, our proposed method maintains a high standard of image quality throughout the stereo generation process, achieving state-of-the-art scores in various quantitative evaluations.

Title: ActFormer: Scalable Collaborative Perception via Active Queries

• Authors: Suozhi Huang, Juexiao Zhang, Yiming Li, Chen Feng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04968
• Pdf URL: https://arxiv.org/pdf/2403.04968
• Copy Paste: [[2403.04968]] ActFormer: Scalable Collaborative Perception via Active Queries(https://arxiv.org/abs/2403.04968)
• Keywords: transformer
• Abstract: Collaborative perception leverages rich visual observations from multiple robots to extend a single robot's perception ability beyond its field of view. Many prior works receive messages broadcast from all collaborators, leading to a scalability challenge when dealing with a large number of robots and sensors. In this work, we aim to address \textit{scalable camera-based collaborative perception} with a Transformer-based architecture. Our key idea is to enable a single robot to intelligently discern the relevance of the collaborators and their associated cameras according to a learned spatial prior. This proactive understanding of the visual features' relevance does not require the transmission of the features themselves, enhancing both communication and computation efficiency. Specifically, we present ActFormer, a Transformer that learns bird's eye view (BEV) representations by using predefined BEV queries to interact with multi-robot multi-camera inputs. Each BEV query can actively select relevant cameras for information aggregation based on pose information, instead of interacting with all cameras indiscriminately. Experiments on the V2X-Sim dataset demonstrate that ActFormer improves the detection performance from 29.89% to 45.15% in terms of AP@0.7 with about 50% fewer queries, showcasing the effectiveness of ActFormer in multi-agent collaborative 3D object detection.

Title: DT-SIM: Property-Based Testing for MPC Security

• Authors: Mako Bates, Joe Near
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.04991
• Pdf URL: https://arxiv.org/pdf/2403.04991
• Copy Paste: [[2403.04991]] DT-SIM: Property-Based Testing for MPC Security(https://arxiv.org/abs/2403.04991)
• Keywords: secure, security
• Abstract: Formal methods for guaranteeing that a protocol satisfies a cryptographic security definition have advanced substantially, but such methods are still labor intensive and the need remains for an automated tool that can positively identify an insecure protocol. In this work, we demonstrate that property-based testing, "run it a bunch of times and see if it breaks", is effective for detecting security bugs in secure protocols. We specifically target Secure Multi-Party Computation (MPC), because formal methods targeting this security definition for bit-model implementations are particularly difficult. Using results from the literature for Probabilistic Programming Languages and statistical inference, we devise a test that can detect various flaws in a bit-level implementation of an MPC protocol. The test is grey-box; it requires only transcripts of randomness consumed by the protocol and of the inputs, outputs, and messages. It successfully detects several different mistakes and biases introduced into two different implementations of the classic GMW protocol. Applied to hundreds of randomly generated protocols, it identifies nearly all of them as insecure. We also include an analysis of the parameters of the test, and discussion of what makes detection of MPC (in)security difficult.

Title: DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation

• Authors: Jiapeng Wang, Chengyu Wang, Tingfeng Cao, Jun Huang, Lianwen Jin
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04997
• Pdf URL: https://arxiv.org/pdf/2403.04997
• Copy Paste: [[2403.04997]] DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation(https://arxiv.org/abs/2403.04997)
• Keywords: diffusion, large language model
• Abstract: We present DiffChat, a novel method to align Large Language Models (LLMs) to "chat" with prompt-as-input Text-to-Image Synthesis (TIS) models (e.g., Stable Diffusion) for interactive image creation. Given a raw prompt/image and a user-specified instruction, DiffChat can effectively make appropriate modifications and generate the target prompt, which can be leveraged to create the target image of high quality. To achieve this, we first collect an instruction-following prompt engineering dataset named InstructPE for the supervised training of DiffChat. Next, we propose a reinforcement learning framework with the feedback of three core criteria for image creation, i.e., aesthetics, user preference, and content integrity. It involves an action-space dynamic modification technique to obtain more relevant positive samples and harder negative samples during the off-policy sampling. Content integrity is also introduced into the value estimation function for further improvement of produced images. Our method can exhibit superior performance than baseline models and strong competitors based on both automatic and human evaluations, which fully demonstrates its effectiveness.

Title: Can't Remember Details in Long Documents? You Need Some R&R

• Authors: Devanshu Agrawal, Shang Gao, Martin Gajek
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05004
• Pdf URL: https://arxiv.org/pdf/2403.05004
• Copy Paste: [[2403.05004]] Can't Remember Details in Long Documents? You Need Some R&R(https://arxiv.org/abs/2403.05004)
• Keywords: large language model
• Abstract: Long-context large language models (LLMs) hold promise for tasks such as question-answering (QA) over long documents, but they tend to miss important information in the middle of context documents (arXiv:2307.03172v3). Here, we introduce $\textit{R&R}$ -- a combination of two novel prompt-based methods called $\textit{reprompting}$ and $\textit{in-context retrieval}$ (ICR) -- to alleviate this effect in document-based QA. In reprompting, we repeat the prompt instructions periodically throughout the context document to remind the LLM of its original task. In ICR, rather than instructing the LLM to answer the question directly, we instruct it to retrieve the top $k$ passage numbers most relevant to the given question, which are then used as an abbreviated context in a second QA prompt. We test R&R with GPT-4 Turbo and Claude-2.1 on documents up to 80k tokens in length and observe a 16-point boost in QA accuracy on average. Our further analysis suggests that R&R improves performance on long document-based QA because it reduces the distance between relevant context and the instructions. Finally, we show that compared to short-context chunkwise methods, R&R enables the use of larger chunks that cost fewer LLM calls and output tokens, while minimizing the drop in accuracy.

Title: DITTO: Dual and Integrated Latent Topologies for Implicit 3D Reconstruction

• Authors: Jaehyeok Shim, Kyungdon Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05005
• Pdf URL: https://arxiv.org/pdf/2403.05005
• Copy Paste: [[2403.05005]] DITTO: Dual and Integrated Latent Topologies for Implicit 3D Reconstruction(https://arxiv.org/abs/2403.05005)
• Keywords: transformer
• Abstract: We propose a novel concept of dual and integrated latent topologies (DITTO in short) for implicit 3D reconstruction from noisy and sparse point clouds. Most existing methods predominantly focus on single latent type, such as point or grid latents. In contrast, the proposed DITTO leverages both point and grid latents (i.e., dual latent) to enhance their strengths, the stability of grid latents and the detail-rich capability of point latents. Concretely, DITTO consists of dual latent encoder and integrated implicit decoder. In the dual latent encoder, a dual latent layer, which is the key module block composing the encoder, refines both latents in parallel, maintaining their distinct shapes and enabling recursive interaction. Notably, a newly proposed dynamic sparse point transformer within the dual latent layer effectively refines point latents. Then, the integrated implicit decoder systematically combines these refined latents, achieving high-fidelity 3D reconstruction and surpassing previous state-of-the-art methods on object- and scene-level datasets, especially in thin and detailed structures.

Title: Provable Multi-Party Reinforcement Learning with Diverse Human Feedback

• Authors: Huiying Zhong, Zhun Deng, Weijie J. Su, Zhiwei Steven Wu, Linjun Zhang
• Subjects: cs.LG, cs.AI, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.05006
• Pdf URL: https://arxiv.org/pdf/2403.05006
• Copy Paste: [[2403.05006]] Provable Multi-Party Reinforcement Learning with Diverse Human Feedback(https://arxiv.org/abs/2403.05006)
• Keywords: fair
• Abstract: Reinforcement learning with human feedback (RLHF) is an emerging paradigm to align models with human preferences. Typically, RLHF aggregates preferences from multiple individuals who have diverse viewpoints that may conflict with each other. Our work \textit{initiates} the theoretical study of multi-party RLHF that explicitly models the diverse preferences of multiple individuals. We show how traditional RLHF approaches can fail since learning a single reward function cannot capture and balance the preferences of multiple individuals. To overcome such limitations, we incorporate meta-learning to learn multiple preferences and adopt different social welfare functions to aggregate the preferences across multiple parties. We focus on the offline learning setting and establish sample complexity bounds, along with efficiency and fairness guarantees, for optimizing diverse social welfare functions such as Nash, Utilitarian, and Leximin welfare functions. Our results show a separation between the sample complexities of multi-party RLHF and traditional single-party RLHF. Furthermore, we consider a reward-free setting, where each individual's preference is no longer consistent with a reward model, and give pessimistic variants of the von Neumann Winner based on offline preference data. Taken together, our work showcases the advantage of multi-party RLHF but also highlights its more demanding statistical complexity.

Title: DiffClass: Diffusion-Based Class Incremental Learning

• Authors: Zichong Meng, Jie Zhang, Changdi Yang, Zheng Zhan, Pu Zhao, Yanzhi WAng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05016
• Pdf URL: https://arxiv.org/pdf/2403.05016
• Copy Paste: [[2403.05016]] DiffClass: Diffusion-Based Class Incremental Learning(https://arxiv.org/abs/2403.05016)
• Keywords: diffusion
• Abstract: Class Incremental Learning (CIL) is challenging due to catastrophic forgetting. On top of that, Exemplar-free Class Incremental Learning is even more challenging due to forbidden access to previous task data. Recent exemplar-free CIL methods attempt to mitigate catastrophic forgetting by synthesizing previous task data. However, they fail to overcome the catastrophic forgetting due to the inability to deal with the significant domain gap between real and synthetic data. To overcome these issues, we propose a novel exemplar-free CIL method. Our method adopts multi-distribution matching (MDM) diffusion models to unify quality and bridge domain gaps among all domains of training data. Moreover, our approach integrates selective synthetic image augmentation (SSIA) to expand the distribution of the training data, thereby improving the model's plasticity and reinforcing the performance of our method's ultimate component, multi-domain adaptation (MDA). With the proposed integrations, our method then reformulates exemplar-free CIL into a multi-domain adaptation problem to implicitly address the domain gap problem to enhance model stability during incremental training. Extensive experiments on benchmark class incremental datasets and settings demonstrate that our method excels previous exemplar-free CIL methods and achieves state-of-the-art performance.

Title: InstructGIE: Towards Generalizable Image Editing

• Authors: Zichong Meng, Changdi Yang, Jun Liu, Hao Tang, Pu Zhao, Yanzhi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05018
• Pdf URL: https://arxiv.org/pdf/2403.05018
• Copy Paste: [[2403.05018]] InstructGIE: Towards Generalizable Image Editing(https://arxiv.org/abs/2403.05018)
• Keywords: robust, diffusion
• Abstract: Recent advances in image editing have been driven by the development of denoising diffusion models, marking a significant leap forward in this field. Despite these advances, the generalization capabilities of recent image editing approaches remain constrained. In response to this challenge, our study introduces a novel image editing framework with enhanced generalization robustness by boosting in-context learning capability and unifying language instruction. This framework incorporates a module specifically optimized for image editing tasks, leveraging the VMamba Block and an editing-shift matching strategy to augment in-context learning. Furthermore, we unveil a selective area-matching technique specifically engineered to address and rectify corrupted details in generated images, such as human facial features, to further improve the quality. Another key innovation of our approach is the integration of a language unification technique, which aligns language embeddings with editing semantics to elevate the quality of image editing. Moreover, we compile the first dataset for image editing with visual prompts and editing instructions that could be used to enhance in-context capability. Trained on this dataset, our methodology not only achieves superior synthesis quality for trained tasks, but also demonstrates robust generalization capability across unseen vision tasks through tailored prompts.

Title: Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

• Authors: Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, Maarten Sap
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05020
• Pdf URL: https://arxiv.org/pdf/2403.05020
• Copy Paste: [[2403.05020]] Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs(https://arxiv.org/abs/2403.05020)
• Keywords: large language model
• Abstract: Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena with LLM-based agents. However, most work has used an omniscient perspective on these simulations (e.g., single LLM to generate all interlocutors), which is fundamentally at odds with the non-omniscient, information asymmetric interactions that humans have. To examine these differences, we develop an evaluation framework to simulate social interactions with LLMs in various settings (omniscient, non-omniscient). Our experiments show that interlocutors simulated omnisciently are much more successful at accomplishing social goals compared to non-omniscient agents, despite the latter being the more realistic setting. Furthermore, we demonstrate that learning from omniscient simulations improves the apparent naturalness of interactions but scarcely enhances goal achievement in cooperative scenarios. Our findings indicate that addressing information asymmetry remains a fundamental challenge for LLM-based agents.

Title: Defending Against Unforeseen Failure Modes with Latent Adversarial Training

• Authors: Stephen Casper, Lennart Schulze, Oam Patel, Dylan Hadfield-Menell
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05030
• Pdf URL: https://arxiv.org/pdf/2403.05030
• Copy Paste: [[2403.05030]] Defending Against Unforeseen Failure Modes with Latent Adversarial Training(https://arxiv.org/abs/2403.05030)
• Keywords: attack, robust
• Abstract: AI systems sometimes exhibit harmful unintended behaviors post-deployment. This is often despite extensive diagnostics and debugging by developers. Minimizing risks from models is challenging because the attack surface is so large. It is not tractable to exhaustively search for inputs that may cause a model to fail. Red-teaming and adversarial training (AT) are commonly used to make AI systems more robust. However, they have not been sufficient to avoid many real-world failure modes that differ from the ones adversarially trained on. In this work, we utilize latent adversarial training (LAT) to defend against vulnerabilities without generating inputs that elicit them. LAT leverages the compressed, abstract, and structured latent representations of concepts that the network actually uses for prediction. We use LAT to remove trojans and defend against held-out classes of adversarial attacks. We show in image classification, text classification, and text generation tasks that LAT usually improves both robustness and performance on clean data relative to AT. This suggests that LAT can be a promising tool for defending against failure modes that are not explicitly identified by developers.

Title: Quantifying Manifolds: Do the manifolds learned by Generative Adversarial Networks converge to the real data manifold

• Authors: Anupam Chaudhuri, Anj Simmons, Mohamed Abdelrazek
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05033
• Pdf URL: https://arxiv.org/pdf/2403.05033
• Copy Paste: [[2403.05033]] Quantifying Manifolds: Do the manifolds learned by Generative Adversarial Networks converge to the real data manifold(https://arxiv.org/abs/2403.05033)
• Keywords: generative
• Abstract: This paper presents our experiments to quantify the manifolds learned by ML models (in our experiment, we use a GAN model) as they train. We compare the manifolds learned at each epoch to the real manifolds representing the real data. To quantify a manifold, we study the intrinsic dimensions and topological features of the manifold learned by the ML model, how these metrics change as we continue to train the model, and whether these metrics convergence over the course of training to the metrics of the real data manifold.

Title: CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model

• Authors: Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, Jun Zhu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05034
• Pdf URL: https://arxiv.org/pdf/2403.05034
• Copy Paste: [[2403.05034]] CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model(https://arxiv.org/abs/2403.05034)
• Keywords: transformer, generative
• Abstract: Feed-forward 3D generative models like the Large Reconstruction Model (LRM) have demonstrated exceptional generation speed. However, the transformer-based methods do not leverage the geometric priors of the triplane component in their architecture, often leading to sub-optimal quality given the limited size of 3D data and slow training. In this work, we present the Convolutional Reconstruction Model (CRM), a high-fidelity feed-forward single image-to-3D generative model. Recognizing the limitations posed by sparse 3D data, we highlight the necessity of integrating geometric priors into network design. CRM builds on the key observation that the visualization of triplane exhibits spatial correspondence of six orthographic images. First, it generates six orthographic view images from a single input image, then feeds these images into a convolutional U-Net, leveraging its strong pixel-level alignment capabilities and significant bandwidth to create a high-resolution triplane. CRM further employs Flexicubes as geometric representation, facilitating direct end-to-end optimization on textured meshes. Overall, our model delivers a high-fidelity textured mesh from an image in just 10 seconds, without any test-time optimization.

Title: Are Human Conversations Special? A Large Language Model Perspective

• Authors: Toshish Jawale, Chaitanya Animesh, Sekhar Vallath, Kartik Talamadupula, Larry Heck
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05045
• Pdf URL: https://arxiv.org/pdf/2403.05045
• Copy Paste: [[2403.05045]] Are Human Conversations Special? A Large Language Model Perspective(https://arxiv.org/abs/2403.05045)
• Keywords: large language model
• Abstract: This study analyzes changes in the attention mechanisms of large language models (LLMs) when used to understand natural conversations between humans (human-human). We analyze three use cases of LLMs: interactions over web content, code, and mathematical texts. By analyzing attention distance, dispersion, and interdependency across these domains, we highlight the unique challenges posed by conversational data. Notably, conversations require nuanced handling of long-term contextual relationships and exhibit higher complexity through their attention patterns. Our findings reveal that while language models exhibit domain-specific attention behaviors, there is a significant gap in their ability to specialize in human conversations. Through detailed attention entropy analysis and t-SNE visualizations, we demonstrate the need for models trained with a diverse array of high-quality conversational data to enhance understanding and generation of human-like dialogue. This research highlights the importance of domain specialization in language models and suggests pathways for future advancement in modeling human conversational nuances.

Title: REPS: Reconstruction-based Point Cloud Sampling

• Authors: Guoqing Zhang, Wenbo Zhao, Jian Liu, Xianming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05047
• Pdf URL: https://arxiv.org/pdf/2403.05047
• Copy Paste: [[2403.05047]] REPS: Reconstruction-based Point Cloud Sampling(https://arxiv.org/abs/2403.05047)
• Keywords: generative
• Abstract: Sampling is widely used in various point cloud tasks as it can effectively reduce resource consumption. Recently, some methods have proposed utilizing neural networks to optimize the sampling process for various task requirements. Currently, deep downsampling methods can be categorized into two main types: generative-based and score-based. Generative-based methods directly generate sampled point clouds using networks, whereas score-based methods assess the importance of points according to specific rules and then select sampled point clouds based on their scores. However, these methods often result in noticeable clustering effects in high-intensity feature areas, compromising their ability to preserve small-scale features and leading to the loss of some structures, thereby affecting the performance of subsequent tasks. In this paper, we propose REPS, a reconstruction-based scoring strategy that evaluates the importance of each vertex by removing and reconstructing them using surrounding vertices. Our reconstruction process comprises point reconstruction and shape reconstruction. The two aforementioned reconstruction methods effectively evaluate the importance of vertices by removing them at different scales for reconstruction. These reconstructions ensure that our method maintains the overall geometric features of the point cloud and avoids disturbing small-scale structures during sampling. Additionally, we propose the Global-Local Fusion Attention (GLFA) module, which aggregates local and global attention features of point clouds, ensuring high-quality reconstruction and sampling effects. Our method outperforms previous approaches in preserving the structural features of the sampled point clouds. Furthermore, abundant experimental results demonstrate the superior performance of our method across various common tasks.

Title: XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution

• Authors: Yunpeng Qu, Kun Yuan, Kai Zhao, Qizhi Xie, Jinhua Hao, Ming Sun, Chao Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05049
• Pdf URL: https://arxiv.org/pdf/2403.05049
• Copy Paste: [[2403.05049]] XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution(https://arxiv.org/abs/2403.05049)
• Keywords: diffusion, generative, large language model
• Abstract: Diffusion-based methods, endowed with a formidable generative prior, have received increasing attention in Image Super-Resolution (ISR) recently. However, as low-resolution (LR) images often undergo severe degradation, it is challenging for ISR models to perceive the semantic and degradation information, resulting in restoration images with incorrect content or unrealistic artifacts. To address these issues, we propose a \textit{Cross-modal Priors for Super-Resolution (XPSR)} framework. Within XPSR, to acquire precise and comprehensive semantic conditions for the diffusion model, cutting-edge Multimodal Large Language Models (MLLMs) are utilized. To facilitate better fusion of cross-modal priors, a \textit{Semantic-Fusion Attention} is raised. To distill semantic-preserved information instead of undesired degradations, a \textit{Degradation-Free Constraint} is attached between LR and its high-resolution (HR) counterpart. Quantitative and qualitative results show that XPSR is capable of generating high-fidelity and high-realism images across synthetic and real-world datasets. Codes will be released at \url{https://github.com/qyp2000/XPSR}.

Title: PrimeComposer: Faster Progressively Combined Diffusion for Image Composition with Attention Steering

• Authors: Yibin Wang, Weizhong Zhang, Jianwei Zheng, Cheng Jin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05053
• Pdf URL: https://arxiv.org/pdf/2403.05053
• Copy Paste: [[2403.05053]] PrimeComposer: Faster Progressively Combined Diffusion for Image Composition with Attention Steering(https://arxiv.org/abs/2403.05053)
• Keywords: diffusion
• Abstract: Image composition involves seamlessly integrating given objects into a specific visual context. The current training-free methods rely on composing attention weights from several samplers to guide the generator. However, since these weights are derived from disparate contexts, their combination leads to coherence confusion in synthesis and loss of appearance information. These issues worsen with their excessive focus on background generation, even when unnecessary in this task. This not only slows down inference but also compromises foreground generation quality. Moreover, these methods introduce unwanted artifacts in the transition area. In this paper, we formulate image composition as a subject-based local editing task, solely focusing on foreground generation. At each step, the edited foreground is combined with the noisy background to maintain scene consistency. To address the remaining issues, we propose PrimeComposer, a faster training-free diffuser that composites the images by well-designed attention steering across different noise levels. This steering is predominantly achieved by our Correlation Diffuser, utilizing its self-attention layers at each step. Within these layers, the synthesized subject interacts with both the referenced object and background, capturing intricate details and coherent relationships. This prior information is encoded into the attention weights, which are then integrated into the self-attention layers of the generator to guide the synthesis process. Besides, we introduce a Region-constrained Cross-Attention to confine the impact of specific subject-related words to desired regions, addressing the unwanted artifacts shown in the prior method thereby further improving the coherence in the transition area. Our method exhibits the fastest inference efficiency and extensive experiments demonstrate our superiority both qualitatively and quantitatively.

Title: MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction

• Authors: Yitao Zhu, Sheng Wang, Mengjie Xu, Zixu Zhuang, Zhixin Wang, Kaidong Wang, Han Zhang, Qian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05055
• Pdf URL: https://arxiv.org/pdf/2403.05055
• Copy Paste: [[2403.05055]] MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction(https://arxiv.org/abs/2403.05055)
• Keywords: robust
• Abstract: Multiple cameras can provide multi-view video coverage of a person. It is necessary to fuse multi-view data, e.g., for subsequent behavioral analysis, while such fusion often relies on calibration of cameras in traditional solutions. However, it is non-trivial to calibrate multiple cameras. In this work, we propose a method to reconstruct 3D human body from multiple uncalibrated camera views. First, we adopt a pre-trained human body encoder to process each individual camera view, such that human body models and parameters can be reconstructed for each view. Next, instead of simply averaging models across views, we train a network to determine the weights of individual views for their fusion, based on the parameters estimated for joints and hands of human body as well as camera positions. Further, we turn to the mesh surface of human body for dynamic fusion, such that facial expression can be seamlessly integrated into the model of human body. Our method has demonstrated superior performance in reconstructing human body upon two public datasets. More importantly, our method can flexibly support ad-hoc deployment of an arbitrary number of cameras, which has significant potential in related applications. We will release source code upon acceptance of the paper.

Title: Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation

• Authors: Yifan Mao, Jian Liu, Xianming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05056
• Pdf URL: https://arxiv.org/pdf/2403.05056
• Copy Paste: [[2403.05056]] Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation(https://arxiv.org/abs/2403.05056)
• Keywords: robust, steal, diffusion
• Abstract: Monocular depth estimation is a crucial task in computer vision. While existing methods have shown impressive results under standard conditions, they often face challenges in reliably performing in scenarios such as low-light or rainy conditions due to the absence of diverse training data. This paper introduces a novel approach named Stealing Stable Diffusion (SSD) prior for robust monocular depth estimation. The approach addresses this limitation by utilizing stable diffusion to generate synthetic images that mimic challenging conditions. Additionally, a self-training mechanism is introduced to enhance the model's depth estimation capability in such challenging environments. To enhance the utilization of the stable diffusion prior further, the DINOv2 encoder is integrated into the depth model architecture, enabling the model to leverage rich semantic priors and improve its scene understanding. Furthermore, a teacher loss is introduced to guide the student models in acquiring meaningful knowledge independently, thus reducing their dependency on the teacher models. The effectiveness of the approach is evaluated on nuScenes and Oxford RobotCar, two challenging public datasets, with the results showing the efficacy of the method. Source code and weights are available at: https://github.com/hitcslj/SSD.

Title: Can we obtain significant success in RST discourse parsing by using Large Language Models?

• Authors: Aru Maekawa, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05065
• Pdf URL: https://arxiv.org/pdf/2403.05065
• Copy Paste: [[2403.05065]] Can we obtain significant success in RST discourse parsing by using Large Language Models?(https://arxiv.org/abs/2403.05065)
• Keywords: large language model
• Abstract: Recently, decoder-only pre-trained large language models (LLMs), with several tens of billion parameters, have significantly impacted a wide range of natural language processing (NLP) tasks. While encoder-only or encoder-decoder pre-trained language models have already proved to be effective in discourse parsing, the extent to which LLMs can perform this task remains an open research question. Therefore, this paper explores how beneficial such LLMs are for Rhetorical Structure Theory (RST) discourse parsing. Here, the parsing process for both fundamental top-down and bottom-up strategies is converted into prompts, which LLMs can work with. We employ Llama 2 and fine-tune it with QLoRA, which has fewer parameters that can be tuned. Experimental results on three benchmark datasets, RST-DT, Instr-DT, and the GUM corpus, demonstrate that Llama 2 with 70 billion parameters in the bottom-up strategy obtained state-of-the-art (SOTA) results with significant differences. Furthermore, our parsers demonstrated generalizability when evaluated on RST-DT, showing that, in spite of being trained with the GUM corpus, it obtained similar performances to those of existing parsers trained with RST-DT.

Title: Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learning

• Authors: Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, Taesup Moon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05066
• Pdf URL: https://arxiv.org/pdf/2403.05066
• Copy Paste: [[2403.05066]] Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learning(https://arxiv.org/abs/2403.05066)
• Keywords: robust
• Abstract: We argue that one of the main obstacles for developing effective Continual Reinforcement Learning (CRL) algorithms is the negative transfer issue occurring when the new task to learn arrives. Through comprehensive experimental validation, we demonstrate that such issue frequently exists in CRL and cannot be effectively addressed by several recent work on mitigating plasticity loss of RL agents. To that end, we develop Reset & Distill (R&D), a simple yet highly effective method, to overcome the negative transfer problem in CRL. R&D combines a strategy of resetting the agent's online actor and critic networks to learn a new task and an offline learning step for distilling the knowledge from the online actor and previous expert's action probabilities. We carried out extensive experiments on long sequence of Meta-World tasks and show that our method consistently outperforms recent baselines, achieving significantly higher success rates across a range of tasks. Our findings highlight the importance of considering negative transfer in CRL and emphasize the need for robust strategies like R&D to mitigate its detrimental effects.

Title: Improving Diffusion-Based Generative Models via Approximated Optimal Transport

• Authors: Daegyu Kim, Jooyoung Choi, Chaehun Shin, Uiwon Hwang, Sungroh Yoon
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05069
• Pdf URL: https://arxiv.org/pdf/2403.05069
• Copy Paste: [[2403.05069]] Improving Diffusion-Based Generative Models via Approximated Optimal Transport(https://arxiv.org/abs/2403.05069)
• Keywords: diffusion, generative
• Abstract: We introduce the Approximated Optimal Transport (AOT) technique, a novel training scheme for diffusion-based generative models. Our approach aims to approximate and integrate optimal transport into the training process, significantly enhancing the ability of diffusion models to estimate the denoiser outputs accurately. This improvement leads to ODE trajectories of diffusion models with lower curvature and reduced truncation errors during sampling. We achieve superior image quality and reduced sampling steps by employing AOT in training. Specifically, we achieve FID scores of 1.88 with just 27 NFEs and 1.73 with 29 NFEs in unconditional and conditional generations, respectively. Furthermore, when applying AOT to train the discriminator for guidance, we establish new state-of-the-art FID scores of 1.68 and 1.58 for unconditional and conditional generations, respectively, each with 29 NFEs. This outcome demonstrates the effectiveness of AOT in enhancing the performance of diffusion models.

Title: Private Count Release: A Simple and Scalable Approach for Private Data Analytics

• Authors: Ryan Rogers
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.05073
• Pdf URL: https://arxiv.org/pdf/2403.05073
• Copy Paste: [[2403.05073]] Private Count Release: A Simple and Scalable Approach for Private Data Analytics(https://arxiv.org/abs/2403.05073)
• Keywords: privacy
• Abstract: We present a data analytics system that ensures accurate counts can be released with differential privacy and minimal onboarding effort while showing instances that outperform other approaches that require more onboarding effort. The primary difference between our proposal and existing approaches is that it does not rely on user contribution bounds over distinct elements, i.e. $\ell_0$-sensitivity bounds, which can significantly bias counts. Contribution bounds for $\ell_0$-sensitivity have been considered as necessary to ensure differential privacy, but we show that this is actually not necessary and can lead to releasing more results that are more accurate. We require minimal hyperparameter tuning and demonstrate results on several publicly available dataset. We hope that this approach will help differential privacy scale to many different data analytics applications.

Title: Benchmarking Large Language Models for Molecule Prediction Tasks

• Authors: Zhiqiang Zhong, Kuangyu Zhou, Davide Mottin
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2403.05075
• Pdf URL: https://arxiv.org/pdf/2403.05075
• Copy Paste: [[2403.05075]] Benchmarking Large Language Models for Molecule Prediction Tasks(https://arxiv.org/abs/2403.05075)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) stand at the forefront of a number of Natural Language Processing (NLP) tasks. Despite the widespread adoption of LLMs in NLP, much of their potential in broader fields remains largely unexplored, and significant limitations persist in their design and implementation. Notably, LLMs struggle with structured data, such as graphs, and often falter when tasked with answering domain-specific questions requiring deep expertise, such as those in biology and chemistry. In this paper, we explore a fundamental question: Can LLMs effectively handle molecule prediction tasks? Rather than pursuing top-tier performance, our goal is to assess how LLMs can contribute to diverse molecule tasks. We identify several classification and regression prediction tasks across six standard molecule datasets. Subsequently, we carefully design a set of prompts to query LLMs on these tasks and compare their performance with existing Machine Learning (ML) models, which include text-based models and those specifically designed for analysing the geometric structure of molecules. Our investigation reveals several key insights: Firstly, LLMs generally lag behind ML models in achieving competitive performance on molecule tasks, particularly when compared to models adept at capturing the geometric structure of molecules, highlighting the constrained ability of LLMs to comprehend graph data. Secondly, LLMs show promise in enhancing the performance of ML models when used collaboratively. Lastly, we engage in a discourse regarding the challenges and promising avenues to harness LLMs for molecule prediction tasks. The code and models are available at https://github.com/zhiqiangzhongddu/LLMaMol.

Title: UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and UnFavOrable Data Sets

• Authors: Youngju Na, Woo Jae Kim, Kyu Beom Han, Suhyeon Ha, Sung-eui Yoon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05086
• Pdf URL: https://arxiv.org/pdf/2403.05086
• Copy Paste: [[2403.05086]] UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and UnFavOrable Data Sets(https://arxiv.org/abs/2403.05086)
• Keywords: robust, transformer
• Abstract: Generalizable neural implicit surface reconstruction aims to obtain an accurate underlying geometry given a limited number of multi-view images from unseen scenes. However, existing methods select only informative and relevant views using predefined scores for training and testing phases. This constraint renders the model impractical in real-world scenarios, where the availability of favorable combinations cannot always be ensured. We introduce and validate a view-combination score to indicate the effectiveness of the input view combination. We observe that previous methods output degenerate solutions under arbitrary and unfavorable sets. Building upon this finding, we propose \textbf{UFORecon}, a robust view-combination generalizable surface reconstruction framework. To achieve this, we apply cross-view matching transformers to model interactions between source images and build correlation frustums to capture global correlations. Additionally, we explicitly encode pairwise feature similarities as view-consistent priors. Our proposed framework significantly outperforms previous methods in terms of view-combination generalizability and also in the conventional generalizable protocol trained with favorable view-combinations. The code is available at \url{https://github.com/Youngju-Na/UFORecon}.

Title: Spectrum Translation for Refinement of Image Generation (STIG) Based on Contrastive Learning and Spectral Filter Profile

• Authors: Seokjun Lee, Seung-Won Jung, Hyunseok Seo
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.05093
• Pdf URL: https://arxiv.org/pdf/2403.05093
• Copy Paste: [[2403.05093]] Spectrum Translation for Refinement of Image Generation (STIG) Based on Contrastive Learning and Spectral Filter Profile(https://arxiv.org/abs/2403.05093)
• Keywords: diffusion, generative
• Abstract: Currently, image generation and synthesis have remarkably progressed with generative models. Despite photo-realistic results, intrinsic discrepancies are still observed in the frequency domain. The spectral discrepancy appeared not only in generative adversarial networks but in diffusion models. In this study, we propose a framework to effectively mitigate the disparity in frequency domain of the generated images to improve generative performance of both GAN and diffusion models. This is realized by spectrum translation for the refinement of image generation (STIG) based on contrastive learning. We adopt theoretical logic of frequency components in various generative networks. The key idea, here, is to refine the spectrum of the generated image via the concept of image-to-image translation and contrastive learning in terms of digital signal processing. We evaluate our framework across eight fake image datasets and various cutting-edge models to demonstrate the effectiveness of STIG. Our framework outperforms other cutting-edges showing significant decreases in FID and log frequency distance of spectrum. We further emphasize that STIG improves image quality by decreasing the spectral anomaly. Additionally, validation results present that the frequency-based deepfake detector confuses more in the case where fake spectrums are manipulated by STIG.

Title: Face2Diffusion for Fast and Editable Face Personalization

• Authors: Kaede Shiohara, Toshihiko Yamasaki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05094
• Pdf URL: https://arxiv.org/pdf/2403.05094
• Copy Paste: [[2403.05094]] Face2Diffusion for Fast and Editable Face Personalization(https://arxiv.org/abs/2403.05094)
• Keywords: diffusion
• Abstract: Face personalization aims to insert specific faces, taken from images, into pretrained text-to-image diffusion models. However, it is still challenging for previous methods to preserve both the identity similarity and editability due to overfitting to training samples. In this paper, we propose Face2Diffusion (F2D) for high-editability face personalization. The core idea behind F2D is that removing identity-irrelevant information from the training pipeline prevents the overfitting problem and improves editability of encoded faces. F2D consists of the following three novel components: 1) Multi-scale identity encoder provides well-disentangled identity features while keeping the benefits of multi-scale information, which improves the diversity of camera poses. 2) Expression guidance disentangles face expressions from identities and improves the controllability of face expressions. 3) Class-guided denoising regularization encourages models to learn how faces should be denoised, which boosts the text-alignment of backgrounds. Extensive experiments on the FaceForensics++ dataset and diverse prompts demonstrate our method greatly improves the trade-off between the identity- and text-fidelity compared to previous state-of-the-art methods.

Title: Exploring the Adversarial Frontier: Quantifying Robustness via Adversarial Hypervolume

• Authors: Ping Guo, Cheng Gong, Xi Lin, Zhiyuan Yang, Qingfu Zhang
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05100
• Pdf URL: https://arxiv.org/pdf/2403.05100
• Copy Paste: [[2403.05100]] Exploring the Adversarial Frontier: Quantifying Robustness via Adversarial Hypervolume(https://arxiv.org/abs/2403.05100)
• Keywords: security, defense, attack, robust
• Abstract: The escalating threat of adversarial attacks on deep learning models, particularly in security-critical fields, has underscored the need for robust deep learning systems. Conventional robustness evaluations have relied on adversarial accuracy, which measures a model's performance under a specific perturbation intensity. However, this singular metric does not fully encapsulate the overall resilience of a model against varying degrees of perturbation. To address this gap, we propose a new metric termed adversarial hypervolume, assessing the robustness of deep learning models comprehensively over a range of perturbation intensities from a multi-objective optimization standpoint. This metric allows for an in-depth comparison of defense mechanisms and recognizes the trivial improvements in robustness afforded by less potent defensive strategies. Additionally, we adopt a novel training algorithm that enhances adversarial robustness uniformly across various perturbation intensities, in contrast to methods narrowly focused on optimizing adversarial accuracy. Our extensive empirical studies validate the effectiveness of the adversarial hypervolume metric, demonstrating its ability to reveal subtle differences in robustness that adversarial accuracy overlooks. This research contributes a new measure of robustness and establishes a standard for assessing and benchmarking the resilience of current and future defensive models against adversarial threats.

Title: Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval

• Authors: Haochen Han, Qinghua Zheng, Guang Dai, Minnan Luo, Jingdong Wang
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.05105
• Pdf URL: https://arxiv.org/pdf/2403.05105
• Copy Paste: [[2403.05105]] Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval(https://arxiv.org/abs/2403.05105)
• Keywords: robust
• Abstract: Collecting well-matched multimedia datasets is crucial for training cross-modal retrieval models. However, in real-world scenarios, massive multimodal data are harvested from the Internet, which inevitably contains Partially Mismatched Pairs (PMPs). Undoubtedly, such semantical irrelevant data will remarkably harm the cross-modal retrieval performance. Previous efforts tend to mitigate this problem by estimating a soft correspondence to down-weight the contribution of PMPs. In this paper, we aim to address this challenge from a new perspective: the potential semantic similarity among unpaired samples makes it possible to excavate useful knowledge from mismatched pairs. To achieve this, we propose L2RM, a general framework based on Optimal Transport (OT) that learns to rematch mismatched pairs. In detail, L2RM aims to generate refined alignments by seeking a minimal-cost transport plan across different modalities. To formalize the rematching idea in OT, first, we propose a self-supervised cost function that automatically learns from explicit similarity-cost mapping relation. Second, we present to model a partial OT problem while restricting the transport among false positives to further boost refined alignments. Extensive experiments on three benchmarks demonstrate our L2RM significantly improves the robustness against PMPs for existing models. The code is available at https://github.com/hhc1997/L2RM.

Title: APPLE: Adversarial Privacy-aware Perturbations on Latent Embedding for Unfairness Mitigation

• Authors: Zikang Xu, Fenghe Tang, Quan Quan, Qingsong Yao, S. Kevin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05114
• Pdf URL: https://arxiv.org/pdf/2403.05114
• Copy Paste: [[2403.05114]] APPLE: Adversarial Privacy-aware Perturbations on Latent Embedding for Unfairness Mitigation(https://arxiv.org/abs/2403.05114)
• Keywords: privacy, fair, segmentation
• Abstract: Ensuring fairness in deep-learning-based segmentors is crucial for health equity. Much effort has been dedicated to mitigating unfairness in the training datasets or procedures. However, with the increasing prevalence of foundation models in medical image analysis, it is hard to train fair models from scratch while preserving utility. In this paper, we propose a novel method, Adversarial Privacy-aware Perturbations on Latent Embedding (APPLE), that can improve the fairness of deployed segmentors by introducing a small latent feature perturber without updating the weights of the original model. By adding perturbation to the latent vector, APPLE decorates the latent vector of segmentors such that no fairness-related features can be passed to the decoder of the segmentors while preserving the architecture and parameters of the segmentor. Experiments on two segmentation datasets and five segmentors (three U-Net-like and two SAM-like) illustrate the effectiveness of our proposed method compared to several unfairness mitigation methods.

Title: CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion

• Authors: Wendi Zheng, Jiayan Teng, Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, Jie Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05121
• Pdf URL: https://arxiv.org/pdf/2403.05121
• Copy Paste: [[2403.05121]] CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion(https://arxiv.org/abs/2403.05121)
• Keywords: diffusion, generative
• Abstract: Recent advancements in text-to-image generative systems have been largely driven by diffusion models. However, single-stage text-to-image diffusion models still face challenges, in terms of computational efficiency and the refinement of image details. To tackle the issue, we propose CogView3, an innovative cascaded framework that enhances the performance of text-to-image diffusion. CogView3 is the first model implementing relay diffusion in the realm of text-to-image generation, executing the task by first creating low-resolution images and subsequently applying relay-based super-resolution. This methodology not only results in competitive text-to-image outputs but also greatly reduces both training and inference costs. Our experimental results demonstrate that CogView3 outperforms SDXL, the current state-of-the-art open-source text-to-image diffusion model, by 77.0\% in human evaluations, all while requiring only about 1/2 of the inference time. The distilled variant of CogView3 achieves comparable performance while only utilizing 1/10 of the inference time by SDXL.

Title: Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis

• Authors: Muxi Chen, Yi Liu, Jian Yi, Changran Xu, Qiuxia Lai, Hongliang Wang, Tsung-Yi Ho, Qiang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05125
• Pdf URL: https://arxiv.org/pdf/2403.05125
• Copy Paste: [[2403.05125]] Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis(https://arxiv.org/abs/2403.05125)
• Keywords: fair, generative
• Abstract: In this paper, we present an empirical study introducing a nuanced evaluation framework for text-to-image (T2I) generative models, applied to human image synthesis. Our framework categorizes evaluations into two distinct groups: first, focusing on image qualities such as aesthetics and realism, and second, examining text conditions through concept coverage and fairness. We introduce an innovative aesthetic score prediction model that assesses the visual appeal of generated images and unveils the first dataset marked with low-quality regions in generated human images to facilitate automatic defect detection. Our exploration into concept coverage probes the model's effectiveness in interpreting and rendering text-based concepts accurately, while our analysis of fairness reveals biases in model outputs, with an emphasis on gender, race, and age. While our study is grounded in human imagery, this dual-faceted approach is designed with the flexibility to be applicable to other forms of image generation, enhancing our understanding of generative models and paving the way to the next generation of more sophisticated, contextually aware, and ethically attuned generative models. We will release our code, the data used for evaluating generative models and the dataset annotated with defective areas soon.

Title: ChatUIE: Exploring Chat-based Unified Information Extraction using Large Language Models

• Authors: Jun Xu, Mengshu Sun, Zhiqiang Zhang, Jun Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05132
• Pdf URL: https://arxiv.org/pdf/2403.05132
• Copy Paste: [[2403.05132]] ChatUIE: Exploring Chat-based Unified Information Extraction using Large Language Models(https://arxiv.org/abs/2403.05132)
• Keywords: extraction, large language model
• Abstract: Recent advancements in large language models have shown impressive performance in general chat. However, their domain-specific capabilities, particularly in information extraction, have certain limitations. Extracting structured information from natural language that deviates from known schemas or instructions has proven challenging for previous prompt-based methods. This motivated us to explore domain-specific modeling in chat-based language models as a solution for extracting structured information from natural language. In this paper, we present ChatUIE, an innovative unified information extraction framework built upon ChatGLM. Simultaneously, reinforcement learning is employed to improve and align various tasks that involve confusing and limited samples. Furthermore, we integrate generation constraints to address the issue of generating elements that are not present in the input. Our experimental results demonstrate that ChatUIE can significantly improve the performance of information extraction with a slight decrease in chatting ability.

Title: ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

• Authors: Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, Gang Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05135
• Pdf URL: https://arxiv.org/pdf/2403.05135
• Copy Paste: [[2403.05135]] ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment(https://arxiv.org/abs/2403.05135)
• Keywords: diffusion, large language model
• Abstract: Diffusion models have demonstrated remarkable performance in the domain of text-to-image generation. However, most widely used models still employ CLIP as their text encoder, which constrains their ability to comprehend dense prompts, encompassing multiple objects, detailed attributes, complex relationships, long-text alignment, etc. In this paper, we introduce an Efficient Large Language Model Adapter, termed ELLA, which equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment without training of either U-Net or LLM. To seamlessly bridge two pre-trained models, we investigate a range of semantic alignment connector designs and propose a novel module, the Timestep-Aware Semantic Connector (TSC), which dynamically extracts timestep-dependent conditions from LLM. Our approach adapts semantic features at different stages of the denoising process, assisting diffusion models in interpreting lengthy and intricate prompts over sampling timesteps. Additionally, ELLA can be readily incorporated with community models and tools to improve their prompt-following capabilities. To assess text-to-image models in dense prompt following, we introduce Dense Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K dense prompts. Extensive experiments demonstrate the superiority of ELLA in dense prompt following compared to state-of-the-art methods, particularly in multiple object compositions involving diverse attributes and relationships.

Title: Improving Diffusion Models for Virtual Try-on

• Authors: Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, Jinwoo Shin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05139
• Pdf URL: https://arxiv.org/pdf/2403.05139
• Copy Paste: [[2403.05139]] Improving Diffusion Models for Virtual Try-on(https://arxiv.org/abs/2403.05139)
• Keywords: diffusion
• Abstract: This paper considers image-based virtual try-on, which renders an image of a person wearing a curated garment, given a pair of images depicting the person and the garment, respectively. Previous works adapt existing exemplar-based inpainting diffusion models for virtual try-on to improve the naturalness of the generated visuals compared to other methods (e.g., GAN-based), but they fail to preserve the identity of the garments. To overcome this limitation, we propose a novel diffusion model that improves garment fidelity and generates authentic virtual try-on images. Our method, coined IDM-VTON, uses two different modules to encode the semantics of garment image; given the base UNet of the diffusion model, 1) the high-level semantics extracted from a visual encoder are fused to the cross-attention layer, and then 2) the low-level features extracted from parallel UNet are fused to the self-attention layer. In addition, we provide detailed textual prompts for both garment and person images to enhance the authenticity of the generated visuals. Finally, we present a customization method using a pair of person-garment images, which significantly improves fidelity and authenticity. Our experimental results show that our method outperforms previous approaches (both diffusion-based and GAN-based) in preserving garment details and generating authentic virtual try-on images, both qualitatively and quantitatively. Furthermore, the proposed customization method demonstrates its effectiveness in a real-world scenario.

Title: Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models

• Authors: Qiuhui Chen, Huping Ye, Yi Hong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05141
• Pdf URL: https://arxiv.org/pdf/2403.05141
• Copy Paste: [[2403.05141]] Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models(https://arxiv.org/abs/2403.05141)
• Keywords: transformer, large language model, segmentation
• Abstract: Understanding 3D medical image volumes is a critical task in the medical domain. However, existing 3D convolution and transformer-based methods have limited semantic understanding of an image volume and also need a large set of volumes for training. Recent advances in multi-modal large language models (MLLMs) provide a new and promising way to understand images with the help of text descriptions. However, most current MLLMs are designed for 2D natural images. To enhance the 3D medical image understanding with 2D MLLMs, we propose a novel pre-training framework called Med3DInsight, which marries existing 3D image encoders with 2D MLLMs and bridges them via a designed Plane-Slice-Aware Transformer (PSAT) module. Extensive experiments demonstrate our SOTA performance on two downstream segmentation and classification tasks, including three public datasets with CT and MRI modalities and comparison to more than ten baselines. Med3DInsight can be easily integrated into any current 3D medical image understanding network and improves its performance by a good margin.

Title: Motion-Guided Dual-Camera Tracker for Low-Cost Skill Evaluation of Gastric Endoscopy

• Authors: Yuelin Zhang, Wanquan Yan, Kim Yan, Chun Ping Lam, Yufu Qiu, Pengyu Zheng, Raymond Shing-Yan Tang, Shing Shin Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05146
• Pdf URL: https://arxiv.org/pdf/2403.05146
• Copy Paste: [[2403.05146]] Motion-Guided Dual-Camera Tracker for Low-Cost Skill Evaluation of Gastric Endoscopy(https://arxiv.org/abs/2403.05146)
• Keywords: robust
• Abstract: Gastric simulators with objective educational feedback have been proven useful for endoscopy training. Existing electronic simulators with feedback are however not commonly adopted due to their high cost. In this work, a motion-guided dual-camera tracker is proposed to provide reliable endoscope tip position feedback at a low cost inside a mechanical simulator for endoscopy skill evaluation, tackling several unique challenges. To address the issue of significant appearance variation of the endoscope tip while keeping dual-camera tracking consistency, the cross-camera mutual template strategy (CMT) is proposed to introduce dynamic transient mutual templates to dual-camera tracking. To alleviate disturbance from large occlusion and distortion by the light source from the endoscope tip, the Mamba-based motion-guided prediction head (MMH) is presented to aggregate visual tracking with historical motion information modeled by the state space model. The proposed tracker was evaluated on datasets captured by low-cost camera pairs during endoscopy procedures performed inside the mechanical simulator. The tracker achieves SOTA performance with robust and consistent tracking on dual cameras. Further downstream evaluation proves that the 3D tip position determined by the proposed tracker enables reliable skill differentiation. The code and dataset will be released upon acceptance.

Title: Towards a Psychology of Machines: Large Language Models Predict Human Memory

• Authors: Markus Huff, Elanur Ulakçı
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05152
• Pdf URL: https://arxiv.org/pdf/2403.05152
• Copy Paste: [[2403.05152]] Towards a Psychology of Machines: Large Language Models Predict Human Memory(https://arxiv.org/abs/2403.05152)
• Keywords: robust, generative, large language model
• Abstract: Large language models (LLMs) are demonstrating remarkable capabilities across various tasks despite lacking a foundation in human cognition. This raises the question: can these models, beyond simply mimicking human language patterns, offer insights into the mechanisms underlying human cognition? This study explores the ability of ChatGPT to predict human performance in a language-based memory task. Building upon theories of text comprehension, we hypothesize that recognizing ambiguous sentences (e.g., "Because Bill drinks wine is never kept in the house") is facilitated by preceding them with contextually relevant information. Participants, both human and ChatGPT, were presented with pairs of sentences. The second sentence was always a garden-path sentence designed to be inherently ambiguous, while the first sentence either provided a fitting (e.g., "Bill has chronic alcoholism") or an unfitting context (e.g., "Bill likes to play golf"). We measured both human's and ChatGPT's ratings of sentence relatedness, ChatGPT's memorability ratings for the garden-path sentences, and humans' spontaneous memory for the garden-path sentences. The results revealed a striking alignment between ChatGPT's assessments and human performance. Sentences deemed more related and assessed as being more memorable by ChatGPT were indeed better remembered by humans, even though ChatGPT's internal mechanisms likely differ significantly from human cognition. This finding, which was confirmed with a robustness check employing synonyms, underscores the potential of generative AI models to predict human performance accurately. We discuss the broader implications of these findings for leveraging LLMs in the development of psychological theories and for gaining a deeper understanding of human cognition.

Title: GSEdit: Efficient Text-Guided Editing of 3D Objects via Gaussian Splatting

• Authors: Francesco Palandra, Andrea Sanchietti, Daniele Baieri, Emanuele Rodolà
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.05154
• Pdf URL: https://arxiv.org/pdf/2403.05154
• Copy Paste: [[2403.05154]] GSEdit: Efficient Text-Guided Editing of 3D Objects via Gaussian Splatting(https://arxiv.org/abs/2403.05154)
• Keywords: diffusion, generative
• Abstract: We present GSEdit, a pipeline for text-guided 3D object editing based on Gaussian Splatting models. Our method enables the editing of the style and appearance of 3D objects without altering their main details, all in a matter of minutes on consumer hardware. We tackle the problem by leveraging Gaussian splatting to represent 3D scenes, and we optimize the model while progressively varying the image supervision by means of a pretrained image-based diffusion model. The input object may be given as a 3D triangular mesh, or directly provided as Gaussians from a generative model such as DreamGaussian. GSEdit ensures consistency across different viewpoints, maintaining the integrity of the original object's information. Compared to previously proposed methods relying on NeRF-like MLP models, GSEdit stands out for its efficiency, making 3D editing tasks much faster. Our editing process is refined via the application of the SDS loss, ensuring that our edits are both precise and accurate. Our comprehensive evaluation demonstrates that GSEdit effectively alters object shape and appearance following the given textual instructions while preserving their coherence and detail.

Title: LanePtrNet: Revisiting Lane Detection as Point Voting and Grouping on Curves

• Authors: Jiayan Cao, Xueyu Zhu, Cheng Qian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05155
• Pdf URL: https://arxiv.org/pdf/2403.05155
• Copy Paste: [[2403.05155]] LanePtrNet: Revisiting Lane Detection as Point Voting and Grouping on Curves(https://arxiv.org/abs/2403.05155)
• Keywords: segmentation
• Abstract: Lane detection plays a critical role in the field of autonomous driving. Prevailing methods generally adopt basic concepts (anchors, key points, etc.) from object detection and segmentation tasks, while these approaches require manual adjustments for curved objects, involve exhaustive searches on predefined anchors, require complex post-processing steps, and may lack flexibility when applied to real-world scenarios.In this paper, we propose a novel approach, LanePtrNet, which treats lane detection as a process of point voting and grouping on ordered sets: Our method takes backbone features as input and predicts a curve-aware centerness, which represents each lane as a point and assigns the most probable center point to it. A novel point sampling method is proposed to generate a set of candidate points based on the votes received. By leveraging features from local neighborhoods, and cross-instance attention score, we design a grouping module that further performs lane-wise clustering between neighboring and seeding points. Furthermore, our method can accommodate a point-based framework, (PointNet++ series, etc.) as an alternative to the backbone. This flexibility enables effortless extension to 3D lane detection tasks. We conduct comprehensive experiments to validate the effectiveness of our proposed approach, demonstrating its superior performance.

Title: On Protecting the Data Privacy of Large Language Models (LLMs): A Survey

• Authors: Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, Xiuzheng Cheng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.05156
• Pdf URL: https://arxiv.org/pdf/2403.05156
• Copy Paste: [[2403.05156]] On Protecting the Data Privacy of Large Language Models (LLMs): A Survey(https://arxiv.org/abs/2403.05156)
• Keywords: privacy, protect, attack, large language model
• Abstract: Large language models (LLMs) are complex artificial intelligence systems capable of understanding, generating and translating human language. They learn language patterns by analyzing large amounts of text data, allowing them to perform writing, conversation, summarizing and other language tasks. When LLMs process and generate large amounts of data, there is a risk of leaking sensitive information, which may threaten data privacy. This paper concentrates on elucidating the data privacy concerns associated with LLMs to foster a comprehensive understanding. Specifically, a thorough investigation is undertaken to delineate the spectrum of data privacy threats, encompassing both passive privacy leakage and active privacy attacks within LLMs. Subsequently, we conduct an assessment of the privacy protection mechanisms employed by LLMs at various stages, followed by a detailed examination of their efficacy and constraints. Finally, the discourse extends to delineate the challenges encountered and outline prospective directions for advancement in the realm of LLM privacy protection.

Title: LVIC: Multi-modality segmentation by Lifting Visual Info as Cue

• Authors: Zichao Dong, Bowen Pang, Xufeng Huang, Hang Ji, Xin Zhan, Junbo Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05159
• Pdf URL: https://arxiv.org/pdf/2403.05159
• Copy Paste: [[2403.05159]] LVIC: Multi-modality segmentation by Lifting Visual Info as Cue(https://arxiv.org/abs/2403.05159)
• Keywords: robust, segmentation
• Abstract: Multi-modality fusion is proven an effective method for 3d perception for autonomous driving. However, most current multi-modality fusion pipelines for LiDAR semantic segmentation have complicated fusion mechanisms. Point painting is a quite straight forward method which directly bind LiDAR points with visual information. Unfortunately, previous point painting like methods suffer from projection error between camera and LiDAR. In our experiments, we find that this projection error is the devil in point painting. As a result of that, we propose a depth aware point painting mechanism, which significantly boosts the multi-modality fusion. Apart from that, we take a deeper look at the desired visual feature for LiDAR to operate semantic segmentation. By Lifting Visual Information as Cue, LVIC ranks 1st on nuScenes LiDAR semantic segmentation benchmark. Our experiments show the robustness and effectiveness. Codes would be make publicly available soon.

Title: MamMIL: Multiple Instance Learning for Whole Slide Images with State Space Models

• Authors: Zijie Fang, Yifeng Wang, Zhi Wang, Jian Zhang, Xiangyang Ji, Yongbing Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05160
• Pdf URL: https://arxiv.org/pdf/2403.05160
• Copy Paste: [[2403.05160]] MamMIL: Multiple Instance Learning for Whole Slide Images with State Space Models(https://arxiv.org/abs/2403.05160)
• Keywords: transformer
• Abstract: Recently, pathological diagnosis, the gold standard for cancer diagnosis, has achieved superior performance by combining the Transformer with the multiple instance learning (MIL) framework using whole slide images (WSIs). However, the giga-pixel nature of WSIs poses a great challenge for the quadratic-complexity self-attention mechanism in Transformer to be applied in MIL. Existing studies usually use linear attention to improve computing efficiency but inevitably bring performance bottlenecks. To tackle this challenge, we propose a MamMIL framework for WSI classification by cooperating the selective structured state space model (i.e., Mamba) with MIL for the first time, enabling the modeling of instance dependencies while maintaining linear complexity. Specifically, to solve the problem that Mamba can only conduct unidirectional one-dimensional (1D) sequence modeling, we innovatively introduce a bidirectional state space model and a 2D context-aware block to enable MamMIL to learn the bidirectional instance dependencies with 2D spatial relationships. Experiments on two datasets show that MamMIL can achieve advanced classification performance with smaller memory footprints than the state-of-the-art MIL frameworks based on the Transformer. The code will be open-sourced if accepted.

Title: Synthetic data generation for system identification: leveraging knowledge transfer from similar systems

• Authors: Dario Piga, Matteo Rufolo, Gabriele Maroni, Manas Mejari, Marco Forgione
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.05164
• Pdf URL: https://arxiv.org/pdf/2403.05164
• Copy Paste: [[2403.05164]] Synthetic data generation for system identification: leveraging knowledge transfer from similar systems(https://arxiv.org/abs/2403.05164)
• Keywords: robust
• Abstract: This paper addresses the challenge of overfitting in the learning of dynamical systems by introducing a novel approach for the generation of synthetic data, aimed at enhancing model generalization and robustness in scenarios characterized by data scarcity. Central to the proposed methodology is the concept of knowledge transfer from systems within the same class. Specifically, synthetic data is generated through a pre-trained meta-model that describes a broad class of systems to which the system of interest is assumed to belong. Training data serves a dual purpose: firstly, as input to the pre-trained meta model to discern the system's dynamics, enabling the prediction of its behavior and thereby generating synthetic output sequences for new input sequences; secondly, in conjunction with synthetic data, to define the loss function used for model estimation. A validation dataset is used to tune a scalar hyper-parameter balancing the relative importance of training and synthetic data in the definition of the loss function. The same validation set can be also used for other purposes, such as early stopping during the training, fundamental to avoid overfitting in case of small-size training datasets. The efficacy of the approach is shown through a numerical example that highlights the advantages of integrating synthetic data into the system identification process.

Title: Unlocking the Potential of Multimodal Unified Discrete Representation through Training-Free Codebook Optimization and Hierarchical Alignment

• Authors: Hai Huang, Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Jieming Zhu, Zhenhua Dong, Zhou Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05168
• Pdf URL: https://arxiv.org/pdf/2403.05168
• Copy Paste: [[2403.05168]] Unlocking the Potential of Multimodal Unified Discrete Representation through Training-Free Codebook Optimization and Hierarchical Alignment(https://arxiv.org/abs/2403.05168)
• Keywords: robust
• Abstract: Recent advances in representation learning have demonstrated the significance of multimodal alignment. The Dual Cross-modal Information Disentanglement (DCID) model, utilizing a unified codebook, shows promising results in achieving fine-grained representation and cross-modal generalization. However, it is still hindered by equal treatment of all channels and neglect of minor event information, resulting in interference from irrelevant channels and limited performance in fine-grained tasks. Thus, in this work, We propose a Training-free Optimization of Codebook (TOC) method to enhance model performance by selecting important channels in the unified space without retraining. Additionally, we introduce the Hierarchical Dual Cross-modal Information Disentanglement (H-DCID) approach to extend information separation and alignment to two levels, capturing more cross-modal details. The experiment results demonstrate significant improvements across various downstream tasks, with TOC contributing to an average improvement of 1.70% for DCID on four tasks, and H-DCID surpassing DCID by an average of 3.64%. The combination of TOC and H-DCID further enhances performance, exceeding DCID by 4.43%. These findings highlight the effectiveness of our methods in facilitating robust and nuanced cross-modal learning, opening avenues for future enhancements. The source code and pre-trained models can be accessed at https://github.com/haihuangcode/TOC_H-DCID.

Title: DiffuLT: How to Make Diffusion Model Useful for Long-tail Recognition

• Authors: Jie Shao, Ke Zhu, Hanxiao Zhang, Jianxin Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05170
• Pdf URL: https://arxiv.org/pdf/2403.05170
• Copy Paste: [[2403.05170]] DiffuLT: How to Make Diffusion Model Useful for Long-tail Recognition(https://arxiv.org/abs/2403.05170)
• Keywords: diffusion, generative
• Abstract: This paper proposes a new pipeline for long-tail (LT) recognition. Instead of re-weighting or re-sampling, we utilize the long-tailed dataset itself to generate a balanced proxy that can be optimized through cross-entropy (CE). Specifically, a randomly initialized diffusion model, trained exclusively on the long-tailed dataset, is employed to synthesize new samples for underrepresented classes. Then, we utilize the inherent information in the original dataset to filter out harmful samples and keep the useful ones. Our strategy, Diffusion model for Long-Tail recognition (DiffuLT), represents a pioneering utilization of generative models in long-tail recognition. DiffuLT achieves state-of-the-art results on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT, surpassing the best competitors with non-trivial margins. Abundant ablations make our pipeline interpretable, too. The whole generation pipeline is done without any external data or pre-trained model weights, making it highly generalizable to real-world long-tailed settings.

Title: Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation

• Authors: Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, Yang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05171
• Pdf URL: https://arxiv.org/pdf/2403.05171
• Copy Paste: [[2403.05171]] Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation(https://arxiv.org/abs/2403.05171)
• Keywords: robust, large language model
• Abstract: We introduce Adversarial Policy Optimization (AdvPO), a novel solution to the pervasive issue of reward over-optimization in Reinforcement Learning from Human Feedback (RLHF) for Large Language Models (LLMs). Over-optimization occurs when a reward model serves as an imperfect proxy for human preference, and RL-driven policy optimization erroneously exploits reward inaccuracies. In this paper, we begin by introducing a lightweight way to quantify uncertainties in rewards, relying solely on the last layer embeddings of the reward model, without the need for computationally expensive reward ensembles. AdvPO then addresses a distributionally robust optimization problem centred around the confidence interval of the reward model's predictions for policy improvement. Through comprehensive experiments on the Anthropic HH and TL;DR summarization datasets, we illustrate the efficacy of AdvPO in mitigating the overoptimization issue, consequently resulting in enhanced performance as evaluated through human-assisted evaluation.

Title: Learning Expressive And Generalizable Motion Features For Face Forgery Detection

• Authors: Jingyi Zhang, Peng Zhang, Jingjing Wang, Di Xie, Shiliang Pu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05172
• Pdf URL: https://arxiv.org/pdf/2403.05172
• Copy Paste: [[2403.05172]] Learning Expressive And Generalizable Motion Features For Face Forgery Detection(https://arxiv.org/abs/2403.05172)
• Keywords: attack
• Abstract: Previous face forgery detection methods mainly focus on appearance features, which may be easily attacked by sophisticated manipulation. Considering the majority of current face manipulation methods generate fake faces based on a single frame, which do not take frame consistency and coordination into consideration, artifacts on frame sequences are more effective for face forgery detection. However, current sequence-based face forgery detection methods use general video classification networks directly, which discard the special and discriminative motion information for face manipulation detection. To this end, we propose an effective sequence-based forgery detection framework based on an existing video classification method. To make the motion features more expressive for manipulation detection, we propose an alternative motion consistency block instead of the original motion features module. To make the learned features more generalizable, we propose an auxiliary anomaly detection block. With these two specially designed improvements, we make a general video classification network achieve promising results on three popular face forgery datasets.

Title: VTruST: Controllable value function based subset selection for Data-Centric Trustworthy AI

• Authors: Soumi Das, Shubhadip Nag, Shreyyash Sharma, Suparna Bhattacharya, Sourangshu Bhattacharya
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05174
• Pdf URL: https://arxiv.org/pdf/2403.05174
• Copy Paste: [[2403.05174]] VTruST: Controllable value function based subset selection for Data-Centric Trustworthy AI(https://arxiv.org/abs/2403.05174)
• Keywords: robust, fair
• Abstract: Trustworthy AI is crucial to the widespread adoption of AI in high-stakes applications with fairness, robustness, and accuracy being some of the key trustworthiness metrics. In this work, we propose a controllable framework for data-centric trustworthy AI (DCTAI)- VTruST, that allows users to control the trade-offs between the different trustworthiness metrics of the constructed training datasets. A key challenge in implementing an efficient DCTAI framework is to design an online value-function-based training data subset selection algorithm. We pose the training data valuation and subset selection problem as an online sparse approximation formulation. We propose a novel online version of the Orthogonal Matching Pursuit (OMP) algorithm for solving this problem. Experimental results show that VTruST outperforms the state-of-the-art baselines on social, image, and scientific datasets. We also show that the data values generated by VTruST can provide effective data-centric explanations for different trustworthiness metrics.

Title: Adversarial Sparse Teacher: Defense Against Distillation-Based Model Stealing Attacks Using Adversarial Examples

• Authors: Eda Yilmaz, Hacer Yalim Keles
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05181
• Pdf URL: https://arxiv.org/pdf/2403.05181
• Copy Paste: [[2403.05181]] Adversarial Sparse Teacher: Defense Against Distillation-Based Model Stealing Attacks Using Adversarial Examples(https://arxiv.org/abs/2403.05181)
• Keywords: protect, defense, attack, steal
• Abstract: Knowledge Distillation (KD) facilitates the transfer of discriminative capabilities from an advanced teacher model to a simpler student model, ensuring performance enhancement without compromising accuracy. It is also exploited for model stealing attacks, where adversaries use KD to mimic the functionality of a teacher model. Recent developments in this domain have been influenced by the Stingy Teacher model, which provided empirical analysis showing that sparse outputs can significantly degrade the performance of student models. Addressing the risk of intellectual property leakage, our work introduces an approach to train a teacher model that inherently protects its logits, influenced by the Nasty Teacher concept. Differing from existing methods, we incorporate sparse outputs of adversarial examples with standard training data to strengthen the teacher's defense against student distillation. Our approach carefully reduces the relative entropy between the original and adversarially perturbed outputs, allowing the model to produce adversarial logits with minimal impact on overall performance. The source codes will be made publicly available soon.

Title: ROUGE-K: Do Your Summaries Have Keywords?

• Authors: Sotaro Takeshita, Simone Paolo Ponzetto, Kai Eckert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05186
• Pdf URL: https://arxiv.org/pdf/2403.05186
• Copy Paste: [[2403.05186]] ROUGE-K: Do Your Summaries Have Keywords?(https://arxiv.org/abs/2403.05186)
• Keywords: transformer
• Abstract: Keywords, that is, content-relevant words in summaries play an important role in efficient information conveyance, making it critical to assess if system-generated summaries contain such informative words during evaluation. However, existing evaluation metrics for extreme summarization models do not pay explicit attention to keywords in summaries, leaving developers ignorant of their presence. To address this issue, we present a keyword-oriented evaluation metric, dubbed ROUGE-K, which provides a quantitative answer to the question of -- \textit{How well do summaries include keywords?} Through the lens of this keyword-aware metric, we surprisingly find that a current strong baseline model often misses essential information in their summaries. Our analysis reveals that human annotators indeed find the summaries with more keywords to be more relevant to the source documents. This is an important yet previously overlooked aspect in evaluating summarization systems. Finally, to enhance keyword inclusion, we propose four approaches for incorporating word importance into a transformer-based model and experimentally show that it enables guiding models to include more keywords while keeping the overall quality. Our code is released at https://github.com/sobamchan/rougek.

Title: CommitBench: A Benchmark for Commit Message Generation

• Authors: Maximilian Schall, Tamara Czinczoll, Gerard de Melo
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.05188
• Pdf URL: https://arxiv.org/pdf/2403.05188
• Copy Paste: [[2403.05188]] CommitBench: A Benchmark for Commit Message Generation(https://arxiv.org/abs/2403.05188)
• Keywords: privacy, transformer
• Abstract: Writing commit messages is a tedious daily task for many software developers, and often remains neglected. Automating this task has the potential to save time while ensuring that messages are informative. A high-quality dataset and an objective benchmark are vital preconditions for solid research and evaluation towards this goal. We show that existing datasets exhibit various problems, such as the quality of the commit selection, small sample sizes, duplicates, privacy issues, and missing licenses for redistribution. This can lead to unusable models and skewed evaluations, where inferior models achieve higher evaluation scores due to biases in the data. We compile a new large-scale dataset, CommitBench, adopting best practices for dataset creation. We sample commits from diverse projects with licenses that permit redistribution and apply our filtering and dataset enhancements to improve the quality of generated commit messages. We use CommitBench to compare existing models and show that other approaches are outperformed by a Transformer model pretrained on source code. We hope to accelerate future research by publishing the source code( https://github.com/Maxscha/commitbench ).

Title: Denoising Autoregressive Representation Learning

• Authors: Yazhe Li, Jorg Bornschein, Ting Chen
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05196
• Pdf URL: https://arxiv.org/pdf/2403.05196
• Copy Paste: [[2403.05196]] Denoising Autoregressive Representation Learning(https://arxiv.org/abs/2403.05196)
• Keywords: diffusion, transformer, generative
• Abstract: In this paper, we explore a new generative approach for learning visual representations. Our method, DARL, employs a decoder-only Transformer to predict image patches autoregressively. We find that training with Mean Squared Error (MSE) alone leads to strong representations. To enhance the image generation ability, we replace the MSE loss with the diffusion objective by using a denoising patch decoder. We show that the learned representation can be improved by using tailored noise schedules and longer training in larger models. Notably, the optimal schedule differs significantly from the typical ones used in standard image diffusion models. Overall, despite its simple architecture, DARL delivers performance remarkably close to state-of-the-art masked prediction models under the fine-tuning protocol. This marks an important step towards a unified model capable of both visual perception and generation, effectively combining the strengths of autoregressive and denoising diffusion models.

Title: TIPS: Threat Sharing Information Platform for Enhanced Security

• Authors: Lakshmi Rama Kiran Pasumarthy, Hisham Ali, William J Buchanan, Jawad Ahmad, Audun Josang, Vasileios Mavroeidis, Mouad Lemoudden
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.05210
• Pdf URL: https://arxiv.org/pdf/2403.05210
• Copy Paste: [[2403.05210]] TIPS: Threat Sharing Information Platform for Enhanced Security(https://arxiv.org/abs/2403.05210)
• Keywords: security, privacy, attack
• Abstract: There is an increasing need to share threat information for the prevention of widespread cyber-attacks. While threat-related information sharing can be conducted through traditional information exchange methods, such as email communications etc., these methods are often weak in terms of their trustworthiness and privacy. Additionally, the absence of a trust infrastructure between different information-sharing domains also poses significant challenges. These challenges include redactment of information, the Right-to-be-forgotten, and access control to the information-sharing elements. These access issues could be related to time bounds, the trusted deletion of data, and the location of accesses. This paper presents an abstraction of a trusted information-sharing process which integrates Attribute-Based Encryption (ABE), Homomorphic Encryption (HE) and Zero Knowledge Proof (ZKP) integrated into a permissioned ledger, specifically Hyperledger Fabric (HLF). It then provides a protocol exchange between two threat-sharing agents that share encrypted messages through a trusted channel. This trusted channel can only be accessed by those trusted in the sharing and could be enabled for each data-sharing element or set up for long-term sharing.

Title: Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answering

• Authors: Hongda Sun, Yuxuan Liu, Chengwei Wu, Haiyu Yan, Cheng Tai, Xin Gao, Shuo Shang, Rui Yan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.05217
• Pdf URL: https://arxiv.org/pdf/2403.05217
• Copy Paste: [[2403.05217]] Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answering(https://arxiv.org/abs/2403.05217)
• Keywords: large language model
• Abstract: Open-domain question answering (ODQA) has emerged as a pivotal research spotlight in information systems. Existing methods follow two main paradigms to collect evidence: (1) The \textit{retrieve-then-read} paradigm retrieves pertinent documents from an external corpus; and (2) the \textit{generate-then-read} paradigm employs large language models (LLMs) to generate relevant documents. However, neither can fully address multifaceted requirements for evidence. To this end, we propose LLMQA, a generalized framework that formulates the ODQA process into three basic steps: query expansion, document selection, and answer generation, combining the superiority of both retrieval-based and generation-based evidence. Since LLMs exhibit their excellent capabilities to accomplish various tasks, we instruct LLMs to play multiple roles as generators, rerankers, and evaluators within our framework, integrating them to collaborate in the ODQA process. Furthermore, we introduce a novel prompt optimization algorithm to refine role-playing prompts and steer LLMs to produce higher-quality evidence and answers. Extensive experimental results on widely used benchmarks (NQ, WebQ, and TriviaQA) demonstrate that LLMQA achieves the best performance in terms of both answer accuracy and evidence quality, showcasing its potential for advancing ODQA research and applications.

Title: Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

• Authors: Liting Lin, Heng Fan, Zhipeng Zhang, Yaowei Wang, Yong Xu, Haibin Ling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05231
• Pdf URL: https://arxiv.org/pdf/2403.05231
• Copy Paste: [[2403.05231]] Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance(https://arxiv.org/abs/2403.05231)
• Keywords: transformer, large language model
• Abstract: Motivated by the Parameter-Efficient Fine-Tuning (PEFT) in large language models, we propose LoRAT, a method that unveils the power of larger Vision Transformers (ViT) for tracking within laboratory-level resources. The essence of our work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency, to the domain of visual tracking. However, unique challenges and potential domain gaps make this transfer not as easy as the first intuition. Firstly, a transformer-based tracker constructs unshared position embedding for template and search image. This poses a challenge for the transfer of LoRA, usually requiring consistency in the design when applied to the pre-trained backbone, to downstream tasks. Secondly, the inductive bias inherent in convolutional heads diminishes the effectiveness of parameter-efficient fine-tuning in tracking models. To overcome these limitations, we first decouple the position embeddings in transformer-based trackers into shared spatial ones and independent type ones. The shared embeddings, which describe the absolute coordinates of multi-resolution images (namely, the template and search images), are inherited from the pre-trained backbones. In contrast, the independent embeddings indicate the sources of each token and are learned from scratch. Furthermore, we design an anchor-free head solely based on a multilayer perceptron (MLP) to adapt PETR, enabling better performance with less computational overhead. With our design, 1) it becomes practical to train trackers with the ViT-g backbone on GPUs with only memory of 25.8GB (batch size of 16); 2) we reduce the training time of the L-224 variant from 35.0 to 10.8 GPU hours; 3) we improve the LaSOT SUC score from 0.703 to 0.743 with the L-224 variant; 4) we fast the inference speed of the L-224 variant from 52 to 119 FPS. Code and models will be released.

Title: Fairness-Aware Interpretable Modeling (FAIM) for Trustworthy Machine Learning in Healthcare

• Authors: Mingxuan Liu, Yilin Ning, Yuhe Ke, Yuqing Shang, Bibhas Chakraborty, Marcus Eng Hock Ong, Roger Vaughan, Nan Liu
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.05235
• Pdf URL: https://arxiv.org/pdf/2403.05235
• Copy Paste: [[2403.05235]] Fairness-Aware Interpretable Modeling (FAIM) for Trustworthy Machine Learning in Healthcare(https://arxiv.org/abs/2403.05235)
• Keywords: fair
• Abstract: The escalating integration of machine learning in high-stakes fields such as healthcare raises substantial concerns about model fairness. We propose an interpretable framework - Fairness-Aware Interpretable Modeling (FAIM), to improve model fairness without compromising performance, featuring an interactive interface to identify a "fairer" model from a set of high-performing models and promoting the integration of data-driven evidence and clinical expertise to enhance contextualized fairness. We demonstrated FAIM's value in reducing sex and race biases by predicting hospital admission with two real-world databases, MIMIC-IV-ED and SGH-ED. We show that for both datasets, FAIM models not only exhibited satisfactory discriminatory performance but also significantly mitigated biases as measured by well-established fairness metrics, outperforming commonly used bias-mitigation methods. Our approach demonstrates the feasibility of improving fairness without sacrificing performance and provides an a modeling mode that invites domain experts to engage, fostering a multidisciplinary effort toward tailored AI fairness.

Title: Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation

• Authors: Junyan Wang, Zhenhong Sun, Zhiyu Tan, Xuanbai Chen, Weihua Chen, Hao Li, Cheng Zhang, Yang Song
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05239
• Pdf URL: https://arxiv.org/pdf/2403.05239
• Copy Paste: [[2403.05239]] Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation(https://arxiv.org/abs/2403.05239)
• Keywords: diffusion
• Abstract: Vanilla text-to-image diffusion models struggle with generating accurate human images, commonly resulting in imperfect anatomies such as unnatural postures or disproportionate limbs.Existing methods address this issue mostly by fine-tuning the model with extra images or adding additional controls -- human-centric priors such as pose or depth maps -- during the image generation phase. This paper explores the integration of these human-centric priors directly into the model fine-tuning stage, essentially eliminating the need for extra conditions at the inference stage. We realize this idea by proposing a human-centric alignment loss to strengthen human-related information from the textual prompts within the cross-attention maps. To ensure semantic detail richness and human structural accuracy during fine-tuning, we introduce scale-aware and step-wise constraints within the diffusion process, according to an in-depth analysis of the cross-attention layer. Extensive experiments show that our method largely improves over state-of-the-art text-to-image models to synthesize high-quality human images based on user-written prompts. Project page: \url{https://hcplayercvpr2024.github.io}.

Title: Hide in Thicket: Generating Imperceptible and Rational Adversarial Perturbations on 3D Point Clouds

• Authors: Tianrui Lou, Xiaojun Jia, Jindong Gu, Li Liu, Siyuan Liang, Bangyan He, Xiaochun Cao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.05247
• Pdf URL: https://arxiv.org/pdf/2403.05247
• Copy Paste: [[2403.05247]] Hide in Thicket: Generating Imperceptible and Rational Adversarial Perturbations on 3D Point Clouds(https://arxiv.org/abs/2403.05247)
• Keywords: attack
• Abstract: Adversarial attack methods based on point manipulation for 3D point cloud classification have revealed the fragility of 3D models, yet the adversarial examples they produce are easily perceived or defended against. The trade-off between the imperceptibility and adversarial strength leads most point attack methods to inevitably introduce easily detectable outlier points upon a successful attack. Another promising strategy, shape-based attack, can effectively eliminate outliers, but existing methods often suffer significant reductions in imperceptibility due to irrational deformations. We find that concealing deformation perturbations in areas insensitive to human eyes can achieve a better trade-off between imperceptibility and adversarial strength, specifically in parts of the object surface that are complex and exhibit drastic curvature changes. Therefore, we propose a novel shape-based adversarial attack method, HiT-ADV, which initially conducts a two-stage search for attack regions based on saliency and imperceptibility scores, and then adds deformation perturbations in each attack region using Gaussian kernel functions. Additionally, HiT-ADV is extendable to physical attack. We propose that by employing benign resampling and benign rigid transformations, we can further enhance physical adversarial strength with little sacrifice to imperceptibility. Extensive experiments have validated the superiority of our method in terms of adversarial and imperceptible properties in both digital and physical spaces. Our code is avaliable at: https://github.com/TRLou/HiT-ADV.

Title: Debiasing Large Visual Language Models

• Authors: Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, Tieniu Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05262
• Pdf URL: https://arxiv.org/pdf/2403.05262
• Copy Paste: [[2403.05262]] Debiasing Large Visual Language Models(https://arxiv.org/abs/2403.05262)
• Keywords: fair, large language model
• Abstract: In the realms of computer vision and natural language processing, Large Vision-Language Models (LVLMs) have become indispensable tools, proficient in generating textual descriptions based on visual inputs. Despite their advancements, our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior rather than the input image. Our empirical experiments underscore the persistence of this bias, as LVLMs often provide confident answers even in the absence of relevant images or given incongruent visual input. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies. Firstly, for tasks such as classification or multi-choice question-answering (QA), we propose a ``calibration'' step through affine transformation to adjust the output distribution. This ``Post-Hoc debias'' approach ensures uniform scores for each answer when the image is absent, serving as an effective regularization technique to alleviate the influence of LLM priors. For more intricate open-ended generation tasks, we extend this method to ``Debias sampling'', drawing inspirations from contrastive decoding methods. Furthermore, our investigation sheds light on the instability of LVLMs across various decoding configurations. Through systematic exploration of different settings, we significantly enhance performance, surpassing reported results and raising concerns about the fairness of existing evaluations. Comprehensive experiments substantiate the effectiveness of our proposed strategies in mitigating biases. These strategies not only prove beneficial in minimizing hallucinations but also contribute to the generation of more helpful and precise illustrations.

Title: ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models

• Authors: Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, Xing Xie, Steven Euijong Whang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05266
• Pdf URL: https://arxiv.org/pdf/2403.05266
• Copy Paste: [[2403.05266]] ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models(https://arxiv.org/abs/2403.05266)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved unprecedented performance in various applications, yet their evaluation remains a critical issue. Existing hallucination benchmarks are either static or lack adjustable complexity for thorough analysis. We contend that utilizing existing relational databases is a promising approach for constructing benchmarks due to their accurate knowledge description via functional dependencies. We propose ERBench to automatically convert any relational database into a benchmark based on the entity-relationship (ER) model. Our key idea is to construct questions using the database schema, records, and functional dependencies such that they can be automatically verified. In addition, we use foreign key constraints to join relations and construct multihop questions, which can be arbitrarily complex and used to debug the intermediate answers of LLMs. Finally, ERBench supports continuous evaluation, multimodal questions, and various prompt engineering techniques. In our experiments, we construct an LLM benchmark using databases of multiple domains and make an extensive comparison of contemporary LLMs. We observe that better LLMs like GPT-4 can handle a larger variety of question types, but are by no means perfect. Also, correct answers do not necessarily imply correct rationales, which is an important evaluation that ERBench does better than other benchmarks for various question types. Code is available at https: //github.com/DILAB-KAIST/ERBench.

Title: Elections in the Post-Quantum Era: Is the Complexity Shield Strong Enough?

• Authors: Šimon Schierreich
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.05273
• Pdf URL: https://arxiv.org/pdf/2403.05273
• Copy Paste: [[2403.05273]] Elections in the Post-Quantum Era: Is the Complexity Shield Strong Enough?(https://arxiv.org/abs/2403.05273)
• Keywords: secure, attack, fair
• Abstract: The election, a cornerstone of democracy, is one of the best-recognizable symbols of democratic governance. Voters' confidence in elections is essential, and these days, we can watch practically in live broadcast what consequences distrust in the fairness of elections may have. From the times of the celebrated Gibbard-Satterthwaite theorem, it is well-known in the social-choice community that most voting systems are vulnerable to the efforts of various players to influence elections. Luckily for us, computing such influence to affect election outcomes is a hard problem from the computational complexity perspective. This intractability is regarded as a ``complexity shield'' that secures voting rules against this malicious behavior. In this work, we consider quantum computers to be a new threat to the complexity shield described above, as they break out of standard computing paradigms and unlock additional computational resources. To this end, we provide an overview of possible attacks on election, discuss the abilities of quantum computing, and chart possible directions for future research in this area.

Title: vSPACE: Voting in a Scalable, Privacy-Aware and Confidential Election

• Authors: Se Elnour, William J Buchanan, Paul Keating, Mwrwan Abubakar, Sirag Elnour
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.05275
• Pdf URL: https://arxiv.org/pdf/2403.05275
• Copy Paste: [[2403.05275]] vSPACE: Voting in a Scalable, Privacy-Aware and Confidential Election(https://arxiv.org/abs/2403.05275)
• Keywords: secure, security, privacy, protect
• Abstract: The vSPACE experimental proof-of-concept (PoC) on the TrueElect[Anon][Creds] protocol presents a novel approach to secure, private, and scalable elections, extending the TrueElect and ElectAnon protocols with the integration of AnonCreds SSI (Self-Sovereign Identity). Such a protocol PoC is situated within a Zero-Trust Architecture (ZTA) and leverages confidential computing, continuous authentication, multi-party computation (MPC), and well-architected framework (WAF) principles to address the challenges of cybersecurity, privacy, and trust over IP (ToIP) protection. Employing a Kubernetes confidential cluster within an Enterprise-Scale Landing Zone (ESLZ), vSPACE integrates Distributed Ledger Technology (DLT) for immutable and certifiable audit trails. The Infrastructure as Code (IaC) model ensures rapid deployment, consistent management, and adherence to security standards, making vSPACE a future-proof solution for digital voting systems.

Title: ContrastDiagnosis: Enhancing Interpretability in Lung Nodule Diagnosis Using Contrastive Learning

• Authors: Chenglong Wang, Yinqiao Yi, Yida Wang, Chengxiu Zhang, Yun Liu, Kensaku Mori, Mei Yuan, Guang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05280
• Pdf URL: https://arxiv.org/pdf/2403.05280
• Copy Paste: [[2403.05280]] ContrastDiagnosis: Enhancing Interpretability in Lung Nodule Diagnosis Using Contrastive Learning(https://arxiv.org/abs/2403.05280)
• Keywords: interpretability, explainability
• Abstract: With the ongoing development of deep learning, an increasing number of AI models have surpassed the performance levels of human clinical practitioners. However, the prevalence of AI diagnostic products in actual clinical practice remains significantly lower than desired. One crucial reason for this gap is the so-called `black box' nature of AI models. Clinicians' distrust of black box models has directly hindered the clinical deployment of AI products. To address this challenge, we propose ContrastDiagnosis, a straightforward yet effective interpretable diagnosis framework. This framework is designed to introduce inherent transparency and provide extensive post-hoc explainability for deep learning model, making them more suitable for clinical medical diagnosis. ContrastDiagnosis incorporates a contrastive learning mechanism to provide a case-based reasoning diagnostic rationale, enhancing the model's transparency and also offers post-hoc interpretability by highlighting similar areas. High diagnostic accuracy was achieved with AUC of 0.977 while maintain a high transparency and explainability.

Title: ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications

• Authors: Sotaro Takeshita, Tommaso Green, Ines Reinig, Kai Eckert, Simone Paolo Ponzetto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05303
• Pdf URL: https://arxiv.org/pdf/2403.05303
• Copy Paste: [[2403.05303]] ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications(https://arxiv.org/abs/2403.05303)
• Keywords: large language model
• Abstract: Extensive efforts in the past have been directed toward the development of summarization datasets. However, a predominant number of these resources have been (semi)-automatically generated, typically through web data crawling, resulting in subpar resources for training and evaluating summarization systems, a quality compromise that is arguably due to the substantial costs associated with generating ground-truth summaries, particularly for diverse languages and specialized domains. To address this issue, we present ACLSum, a novel summarization dataset carefully crafted and evaluated by domain experts. In contrast to previous datasets, ACLSum facilitates multi-aspect summarization of scientific papers, covering challenges, approaches, and outcomes in depth. Through extensive experiments, we evaluate the quality of our resource and the performance of models based on pretrained language models and state-of-the-art large language models (LLMs). Additionally, we explore the effectiveness of extractive versus abstractive summarization within the scholarly domain on the basis of automatically discovered aspects. Our results corroborate previous findings in the general domain and indicate the general superiority of end-to-end aspect-based summarization. Our data is released at https://github.com/sobamchan/aclsum.

Title: RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation

• Authors: Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, Yitao Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05313
• Pdf URL: https://arxiv.org/pdf/2403.05313
• Copy Paste: [[2403.05313]] RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation(https://arxiv.org/abs/2403.05313)
• Keywords: large language model
• Abstract: We explore how iterative revising a chain of thoughts with the help of information retrieval significantly improves large language models' reasoning and generation ability in long-horizon generation tasks, while hugely mitigating hallucination. In particular, the proposed method -- *retrieval-augmented thoughts* (RAT) -- revises each thought step one by one with retrieved information relevant to the task query, the current and the past thought steps, after the initial zero-shot CoT is generated. Applying RAT to GPT-3.5, GPT-4, and CodeLLaMA-7b substantially improves their performances on various long-horizon generation tasks; on average of relatively increasing rating scores by 13.63% on code generation, 16.96% on mathematical reasoning, 19.2% on creative writing, and 42.78% on embodied task planning. The demo page can be found at https://craftjarvis.github.io/RAT

Title: ChatASU: Evoking LLM's Reflexion to Truly Understand Aspect Sentiment in Dialogues

• Authors: Yiding Liu, Jingjing Wang, Jiaming Luo, Tao Zeng, Guodong Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05326
• Pdf URL: https://arxiv.org/pdf/2403.05326
• Copy Paste: [[2403.05326]] ChatASU: Evoking LLM's Reflexion to Truly Understand Aspect Sentiment in Dialogues(https://arxiv.org/abs/2403.05326)
• Keywords: large language model
• Abstract: Aspect Sentiment Understanding (ASU) in interactive scenarios (e.g., Question-Answering and Dialogue) has attracted ever-more interest in recent years and achieved important progresses. However, existing studies on interactive ASU largely ignore the coreference issue for opinion targets (i.e., aspects), while this phenomenon is ubiquitous in interactive scenarios especially dialogues, limiting the ASU performance. Recently, large language models (LLMs) shows the powerful ability to integrate various NLP tasks with the chat paradigm. In this way, this paper proposes a new Chat-based Aspect Sentiment Understanding (ChatASU) task, aiming to explore LLMs' ability in understanding aspect sentiments in dialogue scenarios. Particularly, this ChatASU task introduces a sub-task, i.e., Aspect Chain Reasoning (ACR) task, to address the aspect coreference issue. On this basis, we propose a Trusted Self-reflexion Approach (TSA) with ChatGLM as backbone to ChatASU. Specifically, this TSA treats the ACR task as an auxiliary task to boost the performance of the primary ASU task, and further integrates trusted learning into reflexion mechanisms to alleviate the LLMs-intrinsic factual hallucination problem in TSA. Furthermore, a high-quality ChatASU dataset is annotated to evaluate TSA, and extensive experiments show that our proposed TSA can significantly outperform several state-of-the-art baselines, justifying the effectiveness of TSA to ChatASU and the importance of considering the coreference and hallucination issues in ChatASU.

Title: DiffSF: Diffusion Models for Scene Flow Estimation

• Authors: Yushan Zhang, Bastian Wandt, Maria Magnusson, Michael Felsberg
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05327
• Pdf URL: https://arxiv.org/pdf/2403.05327
• Copy Paste: [[2403.05327]] DiffSF: Diffusion Models for Scene Flow Estimation(https://arxiv.org/abs/2403.05327)
• Keywords: robust, diffusion, transformer
• Abstract: Scene flow estimation is an essential ingredient for a variety of real-world applications, especially for autonomous agents, such as self-driving cars and robots. While recent scene flow estimation approaches achieve a reasonable accuracy, their applicability to real-world systems additionally benefits from a reliability measure. Aiming at improving accuracy while additionally providing an estimate for uncertainty, we propose DiffSF that combines transformer-based scene flow estimation with denoising diffusion models. In the diffusion process, the ground truth scene flow vector field is gradually perturbed by adding Gaussian noise. In the reverse process, starting from randomly sampled Gaussian noise, the scene flow vector field prediction is recovered by conditioning on a source and a target point cloud. We show that the diffusion process greatly increases the robustness of predictions compared to prior approaches resulting in state-of-the-art performance on standard scene flow estimation benchmarks. Moreover, by sampling multiple times with different initial states, the denoising process predicts multiple hypotheses, which enables measuring the output uncertainty, allowing our approach to detect a majority of the inaccurate predictions.

Title: OccFusion: Depth Estimation Free Multi-sensor Fusion for 3D Occupancy Prediction

• Authors: Ji Zhang, Yiran Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05329
• Pdf URL: https://arxiv.org/pdf/2403.05329
• Copy Paste: [[2403.05329]] OccFusion: Depth Estimation Free Multi-sensor Fusion for 3D Occupancy Prediction(https://arxiv.org/abs/2403.05329)
• Keywords: robust
• Abstract: 3D occupancy prediction based on multi-sensor fusion, crucial for a reliable autonomous driving system, enables fine-grained understanding of 3D scenes. Previous fusion-based 3D occupancy predictions relied on depth estimation for processing 2D image features. However, depth estimation is an ill-posed problem, hindering the accuracy and robustness of these methods. Furthermore, fine-grained occupancy prediction demands extensive computational resources. We introduce OccFusion, a multi-modal fusion method free from depth estimation, and a corresponding point cloud sampling algorithm for dense integration of image features. Building on this, we propose an active training method and an active coarse to fine pipeline, enabling the model to adaptively learn more from complex samples and optimize predictions specifically for challenging areas such as small or overlapping objects. The active methods we propose can be naturally extended to any occupancy prediction model. Experiments on the OpenOccupancy benchmark show our method surpasses existing state-of-the-art (SOTA) multi-modal methods in IoU across all categories. Additionally, our model is more efficient during both the training and inference phases, requiring far fewer computational resources. Comprehensive ablation studies demonstrate the effectiveness of our proposed techniques.

Title: Explaining Pre-Trained Language Models with Attribution Scores: An Analysis in Low-Resource Settings

• Authors: Wei Zhou, Heike Adel, Hendrik Schuff, Ngoc Thang Vu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05338
• Pdf URL: https://arxiv.org/pdf/2403.05338
• Copy Paste: [[2403.05338]] Explaining Pre-Trained Language Models with Attribution Scores: An Analysis in Low-Resource Settings(https://arxiv.org/abs/2403.05338)
• Keywords: large language model
• Abstract: Attribution scores indicate the importance of different input parts and can, thus, explain model behaviour. Currently, prompt-based models are gaining popularity, i.a., due to their easier adaptability in low-resource settings. However, the quality of attribution scores extracted from prompt-based models has not been investigated yet. In this work, we address this topic by analyzing attribution scores extracted from prompt-based models w.r.t. plausibility and faithfulness and comparing them with attribution scores extracted from fine-tuned models and large language models. In contrast to previous work, we introduce training size as another dimension into the analysis. We find that using the prompting paradigm (with either encoder-based or decoder-based models) yields more plausible explanations than fine-tuning the models in low-resource settings and Shapley Value Sampling consistently outperforms attention and Integrated Gradients in terms of leading to more plausible and faithful explanations.

Title: Federated Learning Method for Preserving Privacy in Face Recognition System

• Authors: Enoch Solomon, Abraham Woubie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05344
• Pdf URL: https://arxiv.org/pdf/2403.05344
• Copy Paste: [[2403.05344]] Federated Learning Method for Preserving Privacy in Face Recognition System(https://arxiv.org/abs/2403.05344)
• Keywords: secure, privacy, federate, generative
• Abstract: The state-of-the-art face recognition systems are typically trained on a single computer, utilizing extensive image datasets collected from various number of users. However, these datasets often contain sensitive personal information that users may hesitate to disclose. To address potential privacy concerns, we explore the application of federated learning, both with and without secure aggregators, in the context of both supervised and unsupervised face recognition systems. Federated learning facilitates the training of a shared model without necessitating the sharing of individual private data, achieving this by training models on decentralized edge devices housing the data. In our proposed system, each edge device independently trains its own model, which is subsequently transmitted either to a secure aggregator or directly to the central server. To introduce diverse data without the need for data transmission, we employ generative adversarial networks to generate imposter data at the edge. Following this, the secure aggregator or central server combines these individual models to construct a global model, which is then relayed back to the edge devices. Experimental findings based on the CelebA datasets reveal that employing federated learning in both supervised and unsupervised face recognition systems offers dual benefits. Firstly, it safeguards privacy since the original data remains on the edge devices. Secondly, the experimental results demonstrate that the aggregated model yields nearly identical performance compared to the individual models, particularly when the federated model does not utilize a secure aggregator. Hence, our results shed light on the practical challenges associated with privacy-preserving face image training, particularly in terms of the balance between privacy and accuracy.

Title: Multiple Instance Learning with random sampling for Whole Slide Image Classification

• Authors: H. Keshvarikhojasteh, J.P.W. Pluim, M. Veta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05351
• Pdf URL: https://arxiv.org/pdf/2403.05351
• Copy Paste: [[2403.05351]] Multiple Instance Learning with random sampling for Whole Slide Image Classification(https://arxiv.org/abs/2403.05351)
• Keywords: interpretability
• Abstract: In computational pathology, random sampling of patches during training of Multiple Instance Learning (MIL) methods is computationally efficient and serves as a regularization strategy. Despite its promising benefits, questions concerning performance trends for varying sample sizes and its influence on model interpretability remain. Addressing these, we reach an optimal performance enhancement of 1.7% using thirty percent of patches on the CAMELYON16 dataset, and 3.7% with only eight samples on the TUPAC16 dataset. We also find interpretability effects are strongly dataset-dependent, with interpretability impacted on CAMELYON16, while remaining unaffected on TUPAC16. This reinforces that both the performance and interpretability relationships with sampling are closely task-specific. End-to-end training with 1024 samples reveals improvements across both datasets compared to pre-extracted features, further highlighting the potential of this efficient approach.

Title: Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder

• Authors: Jiajie Fan, Amal Trigui, Thomas Bäck, Hao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05352
• Pdf URL: https://arxiv.org/pdf/2403.05352
• Copy Paste: [[2403.05352]] Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder(https://arxiv.org/abs/2403.05352)
• Keywords: generative
• Abstract: A great interest has arisen in using Deep Generative Models (DGM) for generative design. When assessing the quality of the generated designs, human designers focus more on structural plausibility, e.g., no missing component, rather than visual artifacts, e.g., noises in the images. Meanwhile, commonly used metrics such as Fr\'echet Inception Distance (FID) may not evaluate accurately as they tend to penalize visual artifacts instead of structural implausibility. As such, FID might not be suitable to assess the performance of DGMs for a generative design task. In this work, we propose to encode the input designs with a simple Denoising Autoencoder (DAE) and measure the distribution distance in the latent space thereof. We experimentally test our DAE-based metrics with FID and other state-of-the-art metrics on three data sets: compared to FID and some more recent works, e.g., FD$_\text{DINO-V2}$ and topology distance, DAE-based metrics can effectively detect implausible structures and are more consistent with structural inspection by human experts.

Title: The Impact of Quantization on the Robustness of Transformer-based Text Classifiers

• Authors: Seyed Parsa Neshaei, Yasaman Boreshban, Gholamreza Ghassem-Sani, Seyed Abolghasem Mirroshandel
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05365
• Pdf URL: https://arxiv.org/pdf/2403.05365
• Copy Paste: [[2403.05365]] The Impact of Quantization on the Robustness of Transformer-based Text Classifiers(https://arxiv.org/abs/2403.05365)
• Keywords: attack, robust, transformer
• Abstract: Transformer-based models have made remarkable advancements in various NLP areas. Nevertheless, these models often exhibit vulnerabilities when confronted with adversarial attacks. In this paper, we explore the effect of quantization on the robustness of Transformer-based models. Quantization usually involves mapping a high-precision real number to a lower-precision value, aiming at reducing the size of the model at hand. To the best of our knowledge, this work is the first application of quantization on the robustness of NLP models. In our experiments, we evaluate the impact of quantization on BERT and DistilBERT models in text classification using SST-2, Emotion, and MR datasets. We also evaluate the performance of these models against TextFooler, PWWS, and PSO adversarial attacks. Our findings show that quantization significantly improves (by an average of 18.68%) the adversarial accuracy of the models. Furthermore, we compare the effect of quantization versus that of the adversarial training approach on robustness. Our experiments indicate that quantization increases the robustness of the model by 18.80% on average compared to adversarial training without imposing any extra computational overhead during training. Therefore, our results highlight the effectiveness of quantization in improving the robustness of NLP models.

Title: Frequency-Adaptive Dilated Convolution for Semantic Segmentation

• Authors: Linwei Chen, Lin Gu, Ying Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05369
• Pdf URL: https://arxiv.org/pdf/2403.05369
• Copy Paste: [[2403.05369]] Frequency-Adaptive Dilated Convolution for Semantic Segmentation(https://arxiv.org/abs/2403.05369)
• Keywords: segmentation
• Abstract: Dilated convolution, which expands the receptive field by inserting gaps between its consecutive elements, is widely employed in computer vision. In this study, we propose three strategies to improve individual phases of dilated convolution from the view of spectrum analysis. Departing from the conventional practice of fixing a global dilation rate as a hyperparameter, we introduce Frequency-Adaptive Dilated Convolution (FADC), which dynamically adjusts dilation rates spatially based on local frequency components. Subsequently, we design two plug-in modules to directly enhance effective bandwidth and receptive field size. The Adaptive Kernel (AdaKern) module decomposes convolution weights into low-frequency and high-frequency components, dynamically adjusting the ratio between these components on a per-channel basis. By increasing the high-frequency part of convolution weights, AdaKern captures more high-frequency components, thereby improving effective bandwidth. The Frequency Selection (FreqSelect) module optimally balances high- and low-frequency components in feature representations through spatially variant reweighting. It suppresses high frequencies in the background to encourage FADC to learn a larger dilation, thereby increasing the receptive field for an expanded scope. Extensive experiments on segmentation and object detection consistently validate the efficacy of our approach. The code is publicly available at \url{https://github.com/Linwei-Chen/FADC}.

Title: Exploring Robust Features for Few-Shot Object Detection in Satellite Imagery

• Authors: Xavier Bou, Gabriele Facciolo, Rafael Grompone von Gioi, Jean-Michel Morel, Thibaud Ehret
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05381
• Pdf URL: https://arxiv.org/pdf/2403.05381
• Copy Paste: [[2403.05381]] Exploring Robust Features for Few-Shot Object Detection in Satellite Imagery(https://arxiv.org/abs/2403.05381)
• Keywords: robust
• Abstract: The goal of this paper is to perform object detection in satellite imagery with only a few examples, thus enabling users to specify any object class with minimal annotation. To this end, we explore recent methods and ideas from open-vocabulary detection for the remote sensing domain. We develop a few-shot object detector based on a traditional two-stage architecture, where the classification block is replaced by a prototype-based classifier. A large-scale pre-trained model is used to build class-reference embeddings or prototypes, which are compared to region proposal contents for label prediction. In addition, we propose to fine-tune prototypes on available training images to boost performance and learn differences between similar classes, such as aircraft types. We perform extensive evaluations on two remote sensing datasets containing challenging and rare objects. Moreover, we study the performance of both visual and image-text features, namely DINOv2 and CLIP, including two CLIP models specifically tailored for remote sensing applications. Results indicate that visual features are largely superior to vision-language models, as the latter lack the necessary domain-specific vocabulary. Lastly, the developed detector outperforms fully supervised and few-shot methods evaluated on the SIMD and DIOR datasets, despite minimal training parameters.

Title: Generalized Correspondence Matching via Flexible Hierarchical Refinement and Patch Descriptor Distillation

• Authors: Yu Han, Ziwei Long, Yanting Zhang, Jin Wu, Zhijun Fang, Rui Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05388
• Pdf URL: https://arxiv.org/pdf/2403.05388
• Copy Paste: [[2403.05388]] Generalized Correspondence Matching via Flexible Hierarchical Refinement and Patch Descriptor Distillation(https://arxiv.org/abs/2403.05388)
• Keywords: extraction, segmentation
• Abstract: Correspondence matching plays a crucial role in numerous robotics applications. In comparison to conventional hand-crafted methods and recent data-driven approaches, there is significant interest in plug-and-play algorithms that make full use of pre-trained backbone networks for multi-scale feature extraction and leverage hierarchical refinement strategies to generate matched correspondences. The primary focus of this paper is to address the limitations of deep feature matching (DFM), a state-of-the-art (SoTA) plug-and-play correspondence matching approach. First, we eliminate the pre-defined threshold employed in the hierarchical refinement process of DFM by leveraging a more flexible nearest neighbor search strategy, thereby preventing the exclusion of repetitive yet valid matches during the early stages. Our second technical contribution is the integration of a patch descriptor, which extends the applicability of DFM to accommodate a wide range of backbone networks pre-trained across diverse computer vision tasks, including image classification, semantic segmentation, and stereo matching. Taking into account the practical applicability of our method in real-world robotics applications, we also propose a novel patch descriptor distillation strategy to further reduce the computational complexity of correspondence matching. Extensive experiments conducted on three public datasets demonstrate the superior performance of our proposed method. Specifically, it achieves an overall performance in terms of mean matching accuracy of 0.68, 0.92, and 0.95 with respect to the tolerances of 1, 3, and 5 pixels, respectively, on the HPatches dataset, outperforming all other SoTA algorithms. Our source code, demo video, and supplement are publicly available at mias.group/GCM.

Title: DualBEV: CNN is All You Need in View Transformation

• Authors: Peidong Li, Wancheng Shen, Qihao Huang, Dixiao Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05402
• Pdf URL: https://arxiv.org/pdf/2403.05402
• Copy Paste: [[2403.05402]] DualBEV: CNN is All You Need in View Transformation(https://arxiv.org/abs/2403.05402)
• Keywords: robust, transformer
• Abstract: Camera-based Bird's-Eye-View (BEV) perception often struggles between adopting 3D-to-2D or 2D-to-3D view transformation (VT). The 3D-to-2D VT typically employs resource intensive Transformer to establish robust correspondences between 3D and 2D feature, while the 2D-to-3D VT utilizes the Lift-Splat-Shoot (LSS) pipeline for real-time application, potentially missing distant information. To address these limitations, we propose DualBEV, a unified framework that utilizes a shared CNN-based feature transformation incorporating three probabilistic measurements for both strategies. By considering dual-view correspondences in one-stage, DualBEV effectively bridges the gap between these strategies, harnessing their individual strengths. Our method achieves state-of-the-art performance without Transformer, delivering comparable efficiency to the LSS approach, with 55.2% mAP and 63.4% NDS on the nuScenes test set. Code will be released at https://github.com/PeidongLi/DualBEV.

Title: Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting

• Authors: Muyao Wang, Wenchao Chen, Bo Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05406
• Pdf URL: https://arxiv.org/pdf/2403.05406
• Copy Paste: [[2403.05406]] Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting(https://arxiv.org/abs/2403.05406)
• Keywords: transformer, generative
• Abstract: The forecasting of Multivariate Time Series (MTS) has long been an important but challenging task. Due to the non-stationary problem across long-distance time steps, previous studies primarily adopt stationarization method to attenuate the non-stationary problem of the original series for better predictability. However, existing methods always adopt the stationarized series, which ignores the inherent non-stationarity, and has difficulty in modeling MTS with complex distributions due to the lack of stochasticity. To tackle these problems, we first develop a powerful hierarchical probabilistic generative module to consider the non-stationarity and stochastic characteristics within MTS, and then combine it with transformer for a well-defined variational generative dynamic model named Hierarchical Time series Variational Transformer (HTV-Trans), which recovers the intrinsic non-stationary information into temporal dependencies. Being a powerful probabilistic model, HTV-Trans is utilized to learn expressive representations of MTS and applied to forecasting tasks. Extensive experiments on diverse datasets show the efficiency of HTV-Trans on MTS forecasting tasks

Title: SIRST-5K: Exploring Massive Negatives Synthesis with Self-supervised Learning for Robust Infrared Small Target Detection

• Authors: Yahao Lu, Yupei Lin, Han Wu, Xiaoyu Xian, Yukai Shi, Liang Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05416
• Pdf URL: https://arxiv.org/pdf/2403.05416
• Copy Paste: [[2403.05416]] SIRST-5K: Exploring Massive Negatives Synthesis with Self-supervised Learning for Robust Infrared Small Target Detection(https://arxiv.org/abs/2403.05416)
• Keywords: robust, transformer
• Abstract: Single-frame infrared small target (SIRST) detection aims to recognize small targets from clutter backgrounds. Recently, convolutional neural networks have achieved significant advantages in general object detection. With the development of Transformer, the scale of SIRST models is constantly increasing. Due to the limited training samples, performance has not been improved accordingly. The quality, quantity, and diversity of the infrared dataset are critical to the detection of small targets. To highlight this issue, we propose a negative sample augmentation method in this paper. Specifically, a negative augmentation approach is proposed to generate massive negatives for self-supervised learning. Firstly, we perform a sequential noise modeling technology to generate realistic infrared data. Secondly, we fuse the extracted noise with the original data to facilitate diversity and fidelity in the generated data. Lastly, we proposed a negative augmentation strategy to enrich diversity as well as maintain semantic invariance. The proposed algorithm produces a synthetic SIRST-5K dataset, which contains massive pseudo-data and corresponding labels. With a rich diversity of infrared small target data, our algorithm significantly improves the model performance and convergence speed. Compared with other state-of-the-art (SOTA) methods, our method achieves outstanding performance in terms of probability of detection (Pd), false-alarm rate (Fa), and intersection over union (IoU).

Title: Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery

• Authors: Mubashir Noman, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwar, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05419
• Pdf URL: https://arxiv.org/pdf/2403.05419
• Copy Paste: [[2403.05419]] Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery(https://arxiv.org/abs/2403.05419)
• Keywords: transformer
• Abstract: Recent advances in unsupervised learning have demonstrated the ability of large vision models to achieve promising results on downstream tasks by pre-training on large amount of unlabelled data. Such pre-training techniques have also been explored recently in the remote sensing domain due to the availability of large amount of unlabelled data. Different from standard natural image datasets, remote sensing data is acquired from various sensor technologies and exhibit diverse range of scale variations as well as modalities. Existing satellite image pre-training methods either ignore the scale information present in the remote sensing imagery or restrict themselves to use only a single type of data modality. In this paper, we re-visit transformers pre-training and leverage multi-scale information that is effectively utilized with multiple modalities. Our proposed approach, named SatMAE++, performs multi-scale pre-training and utilizes convolution based upsampling blocks to reconstruct the image at higher scales making it extensible to include more scales. Compared to existing works, the proposed SatMAE++ with multi-scale pre-training is equally effective for both optical as well as multi-spectral imagery. Extensive experiments on six datasets reveal the merits of proposed contributions, leading to state-of-the-art performance on all datasets. SatMAE++ achieves mean average precision (mAP) gain of 2.5\% for multi-label classification task on BigEarthNet dataset. Our code and pre-trained models are available at \url{https://github.com/techmn/satmae_pp}.

Title: EVD4UAV: An Altitude-Sensitive Benchmark to Evade Vehicle Detection in UAV

• Authors: Huiming Sun, Jiacheng Guo, Zibo Meng, Tianyun Zhang, Jianwu Fang, Yuewei Lin, Hongkai Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05422
• Pdf URL: https://arxiv.org/pdf/2403.05422
• Copy Paste: [[2403.05422]] EVD4UAV: An Altitude-Sensitive Benchmark to Evade Vehicle Detection in UAV(https://arxiv.org/abs/2403.05422)
• Keywords: security, attack, robust
• Abstract: Vehicle detection in Unmanned Aerial Vehicle (UAV) captured images has wide applications in aerial photography and remote sensing. There are many public benchmark datasets proposed for the vehicle detection and tracking in UAV images. Recent studies show that adding an adversarial patch on objects can fool the well-trained deep neural networks based object detectors, posing security concerns to the downstream tasks. However, the current public UAV datasets might ignore the diverse altitudes, vehicle attributes, fine-grained instance-level annotation in mostly side view with blurred vehicle roof, so none of them is good to study the adversarial patch based vehicle detection attack problem. In this paper, we propose a new dataset named EVD4UAV as an altitude-sensitive benchmark to evade vehicle detection in UAV with 6,284 images and 90,886 fine-grained annotated vehicles. The EVD4UAV dataset has diverse altitudes (50m, 70m, 90m), vehicle attributes (color, type), fine-grained annotation (horizontal and rotated bounding boxes, instance-level mask) in top view with clear vehicle roof. One white-box and two black-box patch based attack methods are implemented to attack three classic deep neural networks based object detectors on EVD4UAV. The experimental results show that these representative attack methods could not achieve the robust altitude-insensitive attack performance.

Title: Part-aware Personalized Segment Anything Model for Patient-Specific Segmentation

• Authors: Chenhui Zhao, Liyue Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05433
• Pdf URL: https://arxiv.org/pdf/2403.05433
• Copy Paste: [[2403.05433]] Part-aware Personalized Segment Anything Model for Patient-Specific Segmentation(https://arxiv.org/abs/2403.05433)
• Keywords: robust, segmentation
• Abstract: Precision medicine, such as patient-adaptive treatments utilizing medical images, poses new challenges for image segmentation algorithms due to (1) the large variability across different patients and (2) the limited availability of annotated data for each patient. In this work, we propose a data-efficient segmentation method to address these challenges, namely Part-aware Personalized Segment Anything Model (P^2SAM). Without any model fine-tuning, P^2SAM enables seamless adaptation to any new patients relying only on one-shot patient-specific data. We introduce a novel part-aware prompt mechanism to select multiple-point prompts based on part-level features of the one-shot data. To further promote the robustness of the selected prompt, we propose a retrieval approach to handle outlier prompts. Extensive experiments demonstrate that P^2SAM improves the performance by +8.0% and +2.0% mean Dice score within two patient-specific segmentation settings, and exhibits impressive generality across different application domains, e.g., +6.4% mIoU on the PerSeg benchmark. Code will be released upon acceptance.

Title: Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs

• Authors: Arijit Nag, Animesh Mukherjee, Niloy Ganguly, Soumen Chakrabarti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05434
• Pdf URL: https://arxiv.org/pdf/2403.05434
• Copy Paste: [[2403.05434]] Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs(https://arxiv.org/abs/2403.05434)
• Keywords: generative, large language model
• Abstract: Large Language Models (LLMs) exhibit impressive zero/few-shot inference and generation quality for high-resource languages(HRLs). A few of them have been trained in low-resource languages (LRLs) and give decent performance. Owing to the prohibitive costs of training LLMs, they are usually used as a network service, with the client charged by the count of input and output tokens. The number of tokens strongly depends on the script and language, as well as the LLM's sub-word vocabulary. We show that LRLs are at a pricing disadvantage, because the well-known LLMs produce more tokens for LRLs than HRLs. This is because most currently popular LLMs are optimized for HRL vocabularies. Our objective is to level the playing field: reduce the cost of processing LRLs in contemporary LLMs while ensuring that predictive and generative qualities are not compromised. As means to reduce the number of tokens processed by the LLM, we consider code-mixing, translation, and transliteration of LRLs to HRLs. We perform an extensive study using the IndicXTREME dataset, covering 15 Indian languages, while using GPT-4 (one of the costliest LLM services released so far) as a commercial LLM. We observe and analyze interesting patterns involving token count, cost,and quality across a multitude of languages and tasks. We show that choosing the best policy to interact with the LLM can reduce cost by 90% while giving better or comparable performance, compared to communicating with the LLM in the original LRL.

Title: VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models

• Authors: Yabo Zhang, Yuxiang Wei, Xianhui Lin, Zheng Hui, Peiran Ren, Xuansong Xie, Xiangyang Ji, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05438
• Pdf URL: https://arxiv.org/pdf/2403.05438
• Copy Paste: [[2403.05438]] VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models(https://arxiv.org/abs/2403.05438)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models (T2I) have demonstrated unprecedented capabilities in creating realistic and aesthetic images. On the contrary, text-to-video diffusion models (T2V) still lag far behind in frame quality and text alignment, owing to insufficient quality and quantity of training videos. In this paper, we introduce VideoElevator, a training-free and plug-and-play method, which elevates the performance of T2V using superior capabilities of T2I. Different from conventional T2V sampling (i.e., temporal and spatial modeling), VideoElevator explicitly decomposes each sampling step into temporal motion refining and spatial quality elevating. Specifically, temporal motion refining uses encapsulated T2V to enhance temporal consistency, followed by inverting to the noise distribution required by T2I. Then, spatial quality elevating harnesses inflated T2I to directly predict less noisy latent, adding more photo-realistic details. We have conducted experiments in extensive prompts under the combination of various T2V and T2I. The results show that VideoElevator not only improves the performance of T2V baselines with foundational T2I, but also facilitates stylistic video synthesis with personalized T2I. Our code is available at https://github.com/YBYBZhang/VideoElevator.

Title: On Practicality of Using ARM TrustZone Trusted Execution Environment for Securing Programmable Logic Controllers

• Authors: Zhiang Li, Daisuke Mashima, Wen Shei Ong, Ertem Esiner, Zbigniew Kalbarczyk, Ee-Chien Chang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.05448
• Pdf URL: https://arxiv.org/pdf/2403.05448
• Copy Paste: [[2403.05448]] On Practicality of Using ARM TrustZone Trusted Execution Environment for Securing Programmable Logic Controllers(https://arxiv.org/abs/2403.05448)
• Keywords: secure, security, attack
• Abstract: Programmable logic controllers (PLCs) are crucial devices for implementing automated control in various industrial control systems (ICS), such as smart power grids, water treatment systems, manufacturing, and transportation systems. Owing to their importance, PLCs are often the target of cyber attackers that are aiming at disrupting the operation of ICS, including the nation's critical infrastructure, by compromising the integrity of control logic execution. While a wide range of cybersecurity solutions for ICS have been proposed, they cannot counter strong adversaries with a foothold on the PLC devices, which could manipulate memory, I/O interface, or PLC logic itself. These days, many ICS devices in the market, including PLCs, run on ARM-based processors, and there is a promising security technology called ARM TrustZone, to offer a Trusted Execution Environment (TEE) on embedded devices. Envisioning that such a hardware-assisted security feature becomes available for ICS devices in the near future, this paper investigates the application of the ARM TrustZone TEE technology for enhancing the security of PLC. Our aim is to evaluate the feasibility and practicality of the TEE-based PLCs through the proof-of-concept design and implementation using open-source software such as OP-TEE and OpenPLC. Our evaluation assesses the performance and resource consumption in real-world ICS configurations, and based on the results, we discuss bottlenecks in the OP-TEE secure OS towards a large-scale ICS and desired changes for its application on ICS devices. Our implementation is made available to public for further study and research.

Title: Attention-guided Feature Distillation for Semantic Segmentation

• Authors: Amir M. Mansourian, Arya Jalali, Rozhan Ahmadi, Shohreh Kasaei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05451
• Pdf URL: https://arxiv.org/pdf/2403.05451
• Copy Paste: [[2403.05451]] Attention-guided Feature Distillation for Semantic Segmentation(https://arxiv.org/abs/2403.05451)
• Keywords: segmentation
• Abstract: In contrast to existing complex methodologies commonly employed for distilling knowledge from a teacher to a student, the pro-posed method showcases the efficacy of a simple yet powerful method for utilizing refined feature maps to transfer attention. The proposed method has proven to be effective in distilling rich information, outperforming existing methods in semantic segmentation as a dense prediction task. The proposed Attention-guided Feature Distillation (AttnFD) method, em-ploys the Convolutional Block Attention Module (CBAM), which refines feature maps by taking into account both channel-specific and spatial information content. By only using the Mean Squared Error (MSE) loss function between the refined feature maps of the teacher and the student,AttnFD demonstrates outstanding performance in semantic segmentation, achieving state-of-the-art results in terms of mean Intersection over Union (mIoU) on the PascalVoc 2012 and Cityscapes datasets. The Code is available at https://github.com/AmirMansurian/AttnFD.

Title: Will GPT-4 Run DOOM?

• Authors: Adrian de Wynter
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05468
• Pdf URL: https://arxiv.org/pdf/2403.05468
• Copy Paste: [[2403.05468]] Will GPT-4 Run DOOM?(https://arxiv.org/abs/2403.05468)
• Keywords: large language model
• Abstract: We show that GPT-4's reasoning and planning capabilities extend to the 1993 first-person shooter Doom. This large language model (LLM) is able to run and play the game with only a few instructions, plus a textual description--generated by the model itself from screenshots--about the state of the game being observed. We find that GPT-4 can play the game to a passable degree: it is able to manipulate doors, combat enemies, and perform pathing. More complex prompting strategies involving multiple model calls provide better results. While further work is required to enable the LLM to play the game as well as its classical, reinforcement learning-based counterparts, we note that GPT-4 required no training, leaning instead on its own reasoning and observational capabilities. We hope our work pushes the boundaries on intelligent, LLM-based agents in video games. We conclude by discussing the ethical implications of our work.

Title: FFSTC: Fongbe to French Speech Translation Corpus

• Authors: D. Fortune Kponou, Frejus A. A. Laleye, Eugene C. Ezin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05488
• Pdf URL: https://arxiv.org/pdf/2403.05488
• Copy Paste: [[2403.05488]] FFSTC: Fongbe to French Speech Translation Corpus(https://arxiv.org/abs/2403.05488)
• Keywords: fair, transformer
• Abstract: In this paper, we introduce the Fongbe to French Speech Translation Corpus (FFSTC) for the first time. This corpus encompasses approximately 31 hours of collected Fongbe language content, featuring both French transcriptions and corresponding Fongbe voice recordings. FFSTC represents a comprehensive dataset compiled through various collection methods and the efforts of dedicated individuals. Furthermore, we conduct baseline experiments using Fairseq's transformer_s and conformer models to evaluate data quality and validity. Our results indicate a score of 8.96 for the transformer_s model and 8.14 for the conformer model, establishing a baseline for the FFSTC corpus.

Title: JointMotion: Joint Self-supervision for Joint Motion Prediction

• Authors: Royden Wagner, Ömer Şahin Taş, Marvin Klemp, Carlos Fernandez
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.05489
• Pdf URL: https://arxiv.org/pdf/2403.05489
• Copy Paste: [[2403.05489]] JointMotion: Joint Self-supervision for Joint Motion Prediction(https://arxiv.org/abs/2403.05489)
• Keywords: transformer
• Abstract: We present JointMotion, a self-supervised learning method for joint motion prediction in autonomous driving. Our method includes a scene-level objective connecting motion and environments, and an instance-level objective to refine learned representations. Our evaluations show that these objectives are complementary and outperform recent contrastive and autoencoding methods as pre-training for joint motion prediction. Furthermore, JointMotion adapts to all common types of environment representations used for motion prediction (i.e., agent-centric, scene-centric, and pairwise relative), and enables effective transfer learning between the Waymo Open Motion and the Argoverse 2 Forecasting datasets. Notably, our method improves the joint final displacement error of Wayformer, Scene Transformer, and HPTR by 3%, 7%, and 11%, respectively.

Title: Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought

• Authors: James Chua, Edward Rees, Hunar Batra, Samuel R. Bowman, Julian Michael, Ethan Perez, Miles Turpin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05518
• Pdf URL: https://arxiv.org/pdf/2403.05518
• Copy Paste: [[2403.05518]] Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought(https://arxiv.org/abs/2403.05518)
• Keywords: explainability
• Abstract: While chain-of-thought prompting (CoT) has the potential to improve the explainability of language model reasoning, it can systematically misrepresent the factors influencing models' behavior--for example, rationalizing answers in line with a user's opinion without mentioning this bias. To mitigate this biased reasoning problem, we introduce bias-augmented consistency training (BCT), an unsupervised fine-tuning scheme that trains models to give consistent reasoning across prompts with and without biasing features. We construct a suite testing nine forms of biased reasoning on seven question-answering tasks, and find that applying BCT to GPT-3.5-Turbo with one bias reduces the rate of biased reasoning by 86% on held-out tasks. Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%. As BCT generalizes to held-out biases and does not require gold labels, this method may hold promise for reducing biased reasoning from as-of-yet unknown biases and on tasks where supervision for ground truth reasoning is unavailable.

Title: Authorship Attribution in Bangla Literature (AABL) via Transfer Learning using ULMFiT

• Authors: Aisha Khatun, Anisur Rahman, Md Saiful Islam, Hemayet Ahmed Chowdhury, Ayesha Tasnim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05519
• Pdf URL: https://arxiv.org/pdf/2403.05519
• Copy Paste: [[2403.05519]] Authorship Attribution in Bangla Literature (AABL) via Transfer Learning using ULMFiT(https://arxiv.org/abs/2403.05519)
• Keywords: security, extraction
• Abstract: Authorship Attribution is the task of creating an appropriate characterization of text that captures the authors' writing style to identify the original author of a given piece of text. With increased anonymity on the internet, this task has become increasingly crucial in various security and plagiarism detection fields. Despite significant advancements in other languages such as English, Spanish, and Chinese, Bangla lacks comprehensive research in this field due to its complex linguistic feature and sentence structure. Moreover, existing systems are not scalable when the number of author increases, and the performance drops for small number of samples per author. In this paper, we propose the use of Average-Stochastic Gradient Descent Weight-Dropped Long Short-Term Memory (AWD-LSTM) architecture and an effective transfer learning approach that addresses the problem of complex linguistic features extraction and scalability for authorship attribution in Bangla Literature (AABL). We analyze the effect of different tokenization, such as word, sub-word, and character level tokenization, and demonstrate the effectiveness of these tokenizations in the proposed model. Moreover, we introduce the publicly available Bangla Authorship Attribution Dataset of 16 authors (BAAD16) containing 17,966 sample texts and 13.4+ million words to solve the standard dataset scarcity problem and release six variations of pre-trained language models for use in any Bangla NLP downstream task. For evaluation, we used our developed BAAD16 dataset as well as other publicly available datasets. Empirically, our proposed model outperformed state-of-the-art models and achieved 99.8% accuracy in the BAAD16 dataset. Furthermore, we showed that the proposed system scales much better even with an increasing number of authors, and performance remains steady despite few training samples.

Title: Probabilistic Image-Driven Traffic Modeling via Remote Sensing

• Authors: Scott Workman, Armin Hadzic
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05521
• Pdf URL: https://arxiv.org/pdf/2403.05521
• Copy Paste: [[2403.05521]] Probabilistic Image-Driven Traffic Modeling via Remote Sensing(https://arxiv.org/abs/2403.05521)
• Keywords: transformer, segmentation
• Abstract: This work addresses the task of modeling spatiotemporal traffic patterns directly from overhead imagery, which we refer to as image-driven traffic modeling. We extend this line of work and introduce a multi-modal, multi-task transformer-based segmentation architecture that can be used to create dense city-scale traffic models. Our approach includes a geo-temporal positional encoding module for integrating geo-temporal context and a probabilistic objective function for estimating traffic speeds that naturally models temporal variations. We evaluate our method extensively using the Dynamic Traffic Speeds (DTS) benchmark dataset and significantly improve the state-of-the-art. Finally, we introduce the DTS++ dataset to support mobility-related location adaptation experiments.

Title: Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapola

• Authors: Yijiang Li, Sucheng Ren, Weipeng Deng, Yuzhi Xu, Ying Gao, Edith Ngai, Haohan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05523
• Pdf URL: https://arxiv.org/pdf/2403.05523
• Copy Paste: [[2403.05523]] Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapola(https://arxiv.org/abs/2403.05523)
• Keywords: data-free, large language model
• Abstract: Out-of-distribution (OOD) generalization is a favorable yet challenging property for deep neural networks. The core challenges lie in the limited availability of source domains that help models learn an invariant representation from the spurious features. Various domain augmentation have been proposed but largely rely on interpolating existing domains and frequently face difficulties in creating truly "novel" domains. Humans, on the other hand, can easily extrapolate novel domains, thus, an intriguing question arises: How can neural networks extrapolate like humans and achieve OOD generalization? We introduce a novel approach to domain extrapolation that leverages reasoning ability and the extensive knowledge encapsulated within large language models (LLMs) to synthesize entirely new domains. Starting with the class of interest, we query the LLMs to extract relevant knowledge for these novel domains. We then bridge the gap between the text-centric knowledge derived from LLMs and the pixel input space of the model using text-to-image generation techniques. By augmenting the training set of domain generalization datasets with high-fidelity, photo-realistic images of these new domains, we achieve significant improvements over all existing methods, as demonstrated in both single and multi-domain generalization across various benchmarks. With the ability to extrapolate any domains for any class, our method has the potential to learn a generalized model for any task without any data. To illustrate, we put forth a much more difficult setting termed, data-free domain generalization, that aims to learn a generalized model in the absence of any collected data. Our empirical findings support the above argument and our methods exhibit commendable performance in this setting, even surpassing the supervised setting by approximately 1-2\% on datasets such as VLCS.

Title: GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM

• Authors: Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, Tuo Zhao
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05527
• Pdf URL: https://arxiv.org/pdf/2403.05527
• Copy Paste: [[2403.05527]] GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM(https://arxiv.org/abs/2403.05527)
• Keywords: generative, large language model
• Abstract: Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to be a memory bound problem, significantly constraining the system throughput. Existing methods rely on dropping unimportant tokens or quantizing all entries uniformly. Such methods, however, often incur high approximation errors to represent the compressed matrices. The autoregressive decoding process further compounds the error of each step, resulting in critical deviation in model generation and deterioration of performance. To tackle this challenge, we propose GEAR, an efficient KV cache compression framework that achieves near-lossless high-ratio compression. GEAR first applies quantization to majority of entries of similar magnitudes to ultra-low precision. It then employs a low rank matrix to approximate the quantization error, and a sparse matrix to remedy individual errors from outlier entries. By adeptly integrating three techniques, GEAR is able to fully exploit their synergistic potentials. Our experiments demonstrate that compared to alternatives, GEAR achieves near-lossless 4-bit KV cache compression with up to 2.38x throughput improvement, while reducing peak-memory size up to 2.29x. Our code is publicly available at https://github.com/HaoKang-Timmy/GEAR.

Title: The Computational Complexity of Learning Gaussian Single-Index Models

• Authors: Alex Damian, Loucas Pillaud-Vivien, Jason D. Lee, Joan Bruna
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.05529
• Pdf URL: https://arxiv.org/pdf/2403.05529
• Copy Paste: [[2403.05529]] The Computational Complexity of Learning Gaussian Single-Index Models(https://arxiv.org/abs/2403.05529)
• Keywords: generative
• Abstract: Single-Index Models are high-dimensional regression problems with planted structure, whereby labels depend on an unknown one-dimensional projection of the input via a generic, non-linear, and potentially non-deterministic transformation. As such, they encompass a broad class of statistical inference tasks, and provide a rich template to study statistical and computational trade-offs in the high-dimensional regime. While the information-theoretic sample complexity to recover the hidden direction is linear in the dimension $d$, we show that computationally efficient algorithms, both within the Statistical Query (SQ) and the Low-Degree Polynomial (LDP) framework, necessarily require $\Omega(d^{k^\star/2})$ samples, where $k^\star$ is a "generative" exponent associated with the model that we explicitly characterize. Moreover, we show that this sample complexity is also sufficient, by establishing matching upper bounds using a partial-trace algorithm. Therefore, our results provide evidence of a sharp computational-to-statistical gap (under both the SQ and LDP class) whenever $k^\star>2$. To complete the study, we provide examples of smooth and Lipschitz deterministic target functions with arbitrarily large generative exponents $k^\star$.

Title: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

• Authors: Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, et al. (619 additional authors not shown)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05530
• Pdf URL: https://arxiv.org/pdf/2403.05530
• Copy Paste: [[2403.05530]] Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context(https://arxiv.org/abs/2403.05530)
• Keywords: large language model
• Abstract: In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks across modalities, improves the state-of-the-art in long-document QA, long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

Title: Tune without Validation: Searching for Learning Rate and Weight Decay on Training Sets

• Authors: Lorenzo Brigato, Stavroula Mougiakakou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05532
• Pdf URL: https://arxiv.org/pdf/2403.05532
• Copy Paste: [[2403.05532]] Tune without Validation: Searching for Learning Rate and Weight Decay on Training Sets(https://arxiv.org/abs/2403.05532)
• Keywords: transformer
• Abstract: We introduce Tune without Validation (Twin), a pipeline for tuning learning rate and weight decay without validation sets. We leverage a recent theoretical framework concerning learning phases in hypothesis space to devise a heuristic that predicts what hyper-parameter (HP) combinations yield better generalization. Twin performs a grid search of trials according to an early-/non-early-stopping scheduler and then segments the region that provides the best results in terms of training loss. Among these trials, the weight norm strongly correlates with predicting generalization. To assess the effectiveness of Twin, we run extensive experiments on 20 image classification datasets and train several families of deep networks, including convolutional, transformer, and feed-forward models. We demonstrate proper HP selection when training from scratch and fine-tuning, emphasizing small-sample scenarios.

Title: Tell, Don't Show!: Language Guidance Eases Transfer Across Domains in Images and Videos

• Authors: Tarun Kalluri, Bodhisattwa Prasad Majumder, Manmohan Chandraker
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05535
• Pdf URL: https://arxiv.org/pdf/2403.05535
• Copy Paste: [[2403.05535]] Tell, Don't Show!: Language Guidance Eases Transfer Across Domains in Images and Videos(https://arxiv.org/abs/2403.05535)
• Keywords: robust
• Abstract: We introduce LaGTran, a novel framework that utilizes readily available or easily acquired text descriptions to guide robust transfer of discriminative knowledge from labeled source to unlabeled target data with domain shifts. While unsupervised adaptation methods have been established to address this problem, they show limitations in handling challenging domain shifts due to their exclusive operation within the pixel-space. Motivated by our observation that semantically richer text modality has more favorable transfer properties, we devise a transfer mechanism to use a source-trained text-classifier to generate predictions on the target text descriptions, and utilize these predictions as supervision for the corresponding images. Our approach driven by language guidance is surprisingly easy and simple, yet significantly outperforms all prior approaches on challenging datasets like GeoNet and DomainNet, validating its extreme effectiveness. To further extend the scope of our study beyond images, we introduce a new benchmark to study ego-exo transfer in videos and find that our language-aided LaGTran yields significant gains in this highly challenging and non-trivial transfer setting. Code, models, and proposed datasets are publicly available at https://tarun005.github.io/lagtran/.