2024-03-04

Title: Prioritizing Informative Features and Examples for Deep Learning from Noisy Data

• Authors: Dongmin Park
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00013
• Pdf URL: https://arxiv.org/pdf/2403.00013
• Copy Paste: [[2403.00013]] Prioritizing Informative Features and Examples for Deep Learning from Noisy Data(https://arxiv.org/abs/2403.00013)
• Keywords: robust, large language model
• Abstract: In this dissertation, we propose a systemic framework that prioritizes informative features and examples to enhance each stage of the development process. Specifically, we prioritize informative features and examples and improve the performance of feature learning, data labeling, and data selection. We first propose an approach to extract only informative features that are inherent to solving a target task by using auxiliary out-of-distribution data. We deactivate the noise features in the target distribution by using that in the out-of-distribution data. Next, we introduce an approach that prioritizes informative examples from unlabeled noisy data in order to reduce the labeling cost of active learning. In order to solve the purity-information dilemma, where an attempt to select informative examples induces the selection of many noisy examples, we propose a meta-model that finds the best balance between purity and informativeness. Lastly, we suggest an approach that prioritizes informative examples from labeled noisy data to preserve the performance of data selection. For labeled image noise data, we propose a data selection method that considers the confidence of neighboring samples to maintain the performance of the state-of-the-art Re-labeling models. For labeled text noise data, we present an instruction selection method that takes diversity into account for ranking the quality of instructions with prompting, thereby enhancing the performance of aligned large language models. Overall, our unified framework induces the deep learning development process robust to noisy data, thereby effectively mitigating noisy features and examples in real-world applications.

Title: Towards Interpreting Multi-Objective Feature Associations

• Authors: Nisha Pillai, Ganga Gireesan, Michael J. Rothrock Jr., Bindu Nanduri, Zhiqian Chen, Mahalingam Ramkumar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00017
• Pdf URL: https://arxiv.org/pdf/2403.00017
• Copy Paste: [[2403.00017]] Towards Interpreting Multi-Objective Feature Associations(https://arxiv.org/abs/2403.00017)
• Keywords: interpretability
• Abstract: Understanding how multiple features are associated and contribute to a specific objective is as important as understanding how each feature contributes to a particular outcome. Interpretability of a single feature in a prediction may be handled in multiple ways; however, in a multi-objective prediction, it is difficult to obtain interpretability of a combination of feature values. To address this issue, we propose an objective specific feature interaction design using multi-labels to find the optimal combination of features in agricultural settings. One of the novel aspects of this design is the identification of a method that integrates feature explanations with global sensitivity analysis in order to ensure combinatorial optimization in multi-objective settings. We have demonstrated in our preliminary experiments that an approximate combination of feature values can be found to achieve the desired outcome using two agricultural datasets: one with pre-harvest poultry farm practices for multi-drug resistance presence, and one with post-harvest poultry farm practices for food-borne pathogens. In our combinatorial optimization approach, all three pathogens are taken into consideration simultaneously to account for the interaction between conditions that favor different types of pathogen growth. These results indicate that explanation-based approaches are capable of identifying combinations of features that reduce pathogen presence in fewer iterations than a baseline.

Title: Transformer-based Parameter Estimation in Statistics

• Authors: Xiaoxin Yin, David S. Yin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.00019
• Pdf URL: https://arxiv.org/pdf/2403.00019
• Copy Paste: [[2403.00019]] Transformer-based Parameter Estimation in Statistics(https://arxiv.org/abs/2403.00019)
• Keywords: transformer
• Abstract: Parameter estimation is one of the most important tasks in statistics, and is key to helping people understand the distribution behind a sample of observations. Traditionally parameter estimation is done either by closed-form solutions (e.g., maximum likelihood estimation for Gaussian distribution), or by iterative numerical methods such as Newton-Raphson method when closed-form solution does not exist (e.g., for Beta distribution). In this paper we propose a transformer-based approach to parameter estimation. Compared with existing solutions, our approach does not require a closed-form solution or any mathematical derivations. It does not even require knowing the probability density function, which is needed by numerical methods. After the transformer model is trained, only a single inference is needed to estimate the parameters of the underlying distribution based on a sample of observations. In the empirical study we compared our approach with maximum likelihood estimation on commonly used distributions such as normal distribution, exponential distribution and beta distribution. It is shown that our approach achieves similar or better accuracy as measured by mean-square-errors.

Title: Auditable Homomorphic-based Decentralized Collaborative AI with Attribute-based Differential Privacy

• Authors: Lo-Yao Yeh, Sheng-Po Tseng, Chia-Hsun Lu, Chih-Ya Shen
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00023
• Pdf URL: https://arxiv.org/pdf/2403.00023
• Copy Paste: [[2403.00023]] Auditable Homomorphic-based Decentralized Collaborative AI with Attribute-based Differential Privacy(https://arxiv.org/abs/2403.00023)
• Keywords: security, privacy, protect, federate
• Abstract: In recent years, the notion of federated learning (FL) has led to the new paradigm of distributed artificial intelligence (AI) with privacy preservation. However, most current FL systems suffer from data privacy issues due to the requirement of a trusted third party. Although some previous works introduce differential privacy to protect the data, however, it may also significantly deteriorate the model performance. To address these issues, we propose a novel decentralized collaborative AI framework, named Auditable Homomorphic-based Decentralised Collaborative AI (AerisAI), to improve security with homomorphic encryption and fine-grained differential privacy. Our proposed AerisAI directly aggregates the encrypted parameters with a blockchain-based smart contract to get rid of the need of a trusted third party. We also propose a brand-new concept for eliminating the negative impacts of differential privacy for model performance. Moreover, the proposed AerisAI also provides the broadcast-aware group key management based on ciphertext-policy attribute-based encryption (CPABE) to achieve fine-grained access control based on different service-level agreements. We provide a formal theoretical analysis of the proposed AerisAI as well as the functionality comparison with the other baselines. We also conduct extensive experiments on real datasets to evaluate the proposed approach. The experimental results indicate that our proposed AerisAI significantly outperforms the other state-of-the-art baselines.

Title: On the Challenges and Opportunities in Generative AI

• Authors: Laura Manduchi, Kushagra Pandey, Robert Bamler, Ryan Cotterell, Sina Däubener, Sophie Fellenz, Asja Fischer, Thomas Gärtner, Matthias Kirchler, Marius Kloft, Yingzhen Li, Christoph Lippert, Gerard de Melo, Eric Nalisnick, Björn Ommer, Rajesh Ranganath, Maja Rudolph, Karen Ullrich, Guy Van den Broeck, Julia E Vogt, Yixin Wang, Florian Wenzel, Frank Wood, Stephan Mandt, Vincent Fortuin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00025
• Pdf URL: https://arxiv.org/pdf/2403.00025
• Copy Paste: [[2403.00025]] On the Challenges and Opportunities in Generative AI(https://arxiv.org/abs/2403.00025)
• Keywords: robust, generative
• Abstract: The field of deep generative modeling has grown rapidly and consistently over the years. With the availability of massive amounts of training data coupled with advances in scalable unsupervised learning paradigms, recent large-scale generative models show tremendous promise in synthesizing high-resolution images and text, as well as structured data such as videos and molecules. However, we argue that current large-scale generative AI models do not sufficiently address several fundamental issues that hinder their widespread adoption across domains. In this work, we aim to identify key unresolved challenges in modern generative AI paradigms that should be tackled to further enhance their capabilities, versatility, and reliability. By identifying these challenges, we aim to provide researchers with valuable insights for exploring fruitful research directions, thereby fostering the development of more robust and accessible generative AI solutions.

Title: Learning to Deliver: a Foundation Model for the Montreal Capacitated Vehicle Routing Problem

• Authors: Samuel J. K. Chin, Matthias Winkenbach, Akash Srivastava
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2403.00026
• Pdf URL: https://arxiv.org/pdf/2403.00026
• Copy Paste: [[2403.00026]] Learning to Deliver: a Foundation Model for the Montreal Capacitated Vehicle Routing Problem(https://arxiv.org/abs/2403.00026)
• Keywords: transformer, large language model
• Abstract: In this paper, we present the Foundation Model for the Montreal Capacitated Vehicle Routing Problem (FM-MCVRP), a novel Deep Learning (DL) model that approximates high-quality solutions to a variant of the Capacitated Vehicle Routing Problem (CVRP) that characterizes many real-world applications. The so-called Montreal Capacitated Vehicle Routing Problem (MCVRP), first formally described by Bengio et al. (2021), is defined on a fixed and finite graph, which is analogous to a city. Each MCVRP instance is essentially the sub-graph connecting a randomly sampled subset of the nodes in the fixed graph, which represent a set of potential addresses in a real-world delivery problem on a given day. Our work exploits this problem structure to frame the MCVRP as an analogous Natural Language Processing (NLP) task. Specifically, we leverage a Transformer architecture embedded in a Large Language Model (LLM) framework to train our model in a supervised manner on computationally inexpensive, sub-optimal MCVRP solutions obtained algorithmically. Through comprehensive computational experiments, we show that FM-MCVRP produces better MCVRP solutions than the training data and generalizes to larger sized problem instances not seen during training. Even when compared to near-optimal solutions from state-of-the-art heuristics, FM-MCVRP yields competitive results despite being trained on inferior data. For instance, for 400-customer problems, FM-MCVRP solutions on average fall within 2% of the benchmark. Our results further demonstrate that unlike prior works in the literature, FM-MCVRP is a unified model, which performs consistently and reliably on a range of problem instance sizes and parameter values such as the vehicle capacity.

Title: Lower Bounds for Differential Privacy Under Continual Observation and Online Threshold Queries

• Authors: Edith Cohen, Xin Lyu, Jelani Nelson, Tamás Sarlós, Uri Stemmer
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00028
• Pdf URL: https://arxiv.org/pdf/2403.00028
• Copy Paste: [[2403.00028]] Lower Bounds for Differential Privacy Under Continual Observation and Online Threshold Queries(https://arxiv.org/abs/2403.00028)
• Keywords: privacy
• Abstract: One of the most basic problems for studying the "price of privacy over time" is the so called private counter problem, introduced by Dwork et al. (2010) and Chan et al. (2010). In this problem, we aim to track the number of events that occur over time, while hiding the existence of every single event. More specifically, in every time step $t\in[T]$ we learn (in an online fashion) that $\Delta_t\geq 0$ new events have occurred, and must respond with an estimate $n_t\approx\sum_{j=1}^t \Delta_j$. The privacy requirement is that all of the outputs together, across all time steps, satisfy event level differential privacy. The main question here is how our error needs to depend on the total number of time steps $T$ and the total number of events $n$. Dwork et al. (2015) showed an upper bound of $O\left(\log(T)+\log^2(n)\right)$, and Henzinger et al. (2023) showed a lower bound of $\Omega\left(\min\{\log n, \log T\}\right)$. We show a new lower bound of $\Omega\left(\min\{n,\log T\}\right)$, which is tight w.r.t. the dependence on $T$, and is tight in the sparse case where $\log^2 n=O(\log T)$. Our lower bound has the following implications: $\bullet$ We show that our lower bound extends to the "online thresholds problem", where the goal is to privately answer many "quantile queries" when these queries are presented one-by-one. This resolves an open question of Bun et al. (2017). $\bullet$ Our lower bound implies, for the first time, a separation between the number of mistakes obtainable by a private online learner and a non-private online learner. This partially resolves a COLT'22 open question published by Sanyal and Ramponi. $\bullet$ Our lower bound also yields the first separation between the standard model of private online learning and a recently proposed relaxed variant of it, called private online prediction.

Title: Global and Local Prompts Cooperation via Optimal Transport for Federated Learning

• Authors: Hongxia Li, Wei Huang, Jingya Wang, Ye Shi
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.00041
• Pdf URL: https://arxiv.org/pdf/2403.00041
• Copy Paste: [[2403.00041]] Global and Local Prompts Cooperation via Optimal Transport for Federated Learning(https://arxiv.org/abs/2403.00041)
• Keywords: federate
• Abstract: Prompt learning in pretrained visual-language models has shown remarkable flexibility across various downstream tasks. Leveraging its inherent lightweight nature, recent research attempted to integrate the powerful pretrained models into federated learning frameworks to simultaneously reduce communication costs and promote local training on insufficient data. Despite these efforts, current federated prompt learning methods lack specialized designs to systematically address severe data heterogeneities, e.g., data distribution with both label and feature shifts involved. To address this challenge, we present Federated Prompts Cooperation via Optimal Transport (FedOTP), which introduces efficient collaborative prompt learning strategies to capture diverse category traits on a per-client basis. Specifically, for each client, we learn a global prompt to extract consensus knowledge among clients, and a local prompt to capture client-specific category characteristics. Unbalanced Optimal Transport is then employed to align local visual features with these prompts, striking a balance between global consensus and local personalization. Extensive experiments on datasets with various types of heterogeneities have demonstrated that our FedOTP outperforms the state-of-the-art methods.

Title: Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization

• Authors: Md Tahmid Rahman Laskar, Elena Khasanova, Xue-Yong Fu, Cheng Chen, Shashi Bhushan TN
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00067
• Pdf URL: https://arxiv.org/pdf/2403.00067
• Copy Paste: [[2403.00067]] Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization(https://arxiv.org/abs/2403.00067)
• Keywords: large language model
• Abstract: This work focuses on the task of query-based meeting summarization in which the summary of a context (meeting transcript) is generated in response to a specific query. When using Large Language Models (LLMs) for this task, a new call to the LLM inference endpoint/API is required for each new query even if the context stays the same. However, repeated calls to the LLM inference endpoints would significantly increase the costs of using them in production, making LLMs impractical for many real-world use cases. To address this problem, in this paper, we investigate whether combining the queries for the same input context in a single prompt to minimize repeated calls can be successfully used in meeting summarization. In this regard, we conduct extensive experiments by comparing the performance of various popular LLMs: GPT-4, PaLM-2, LLaMA-2, Mistral, and FLAN-T5 in single-query and multi-query settings. We observe that while most LLMs tend to respond to the multi-query instructions, almost all of them (except GPT-4), even after fine-tuning, could not properly generate the response in the required output format. We conclude that while multi-query prompting could be useful to optimize the inference costs by reducing calls to the inference endpoints/APIs for the task of meeting summarization, this capability to reliably generate the response in the expected format is only limited to certain LLMs.

Title: Resonance RoPE: Improving Context Length Generalization of Large Language Models

• Authors: Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, Bang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00071
• Pdf URL: https://arxiv.org/pdf/2403.00071
• Copy Paste: [[2403.00071]] Resonance RoPE: Improving Context Length Generalization of Large Language Models(https://arxiv.org/abs/2403.00071)
• Keywords: robust, transformer, large language model
• Abstract: This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE), where models pre-trained on shorter sequences face difficulty with out-of-distribution (OOD) token positions in longer sequences. We introduce Resonance RoPE, a novel approach designed to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions, significantly improving the model performance without additional online computational costs. Furthermore, we present PosGen, a new synthetic benchmark specifically designed for fine-grained behavior analysis in TSTL scenarios, aiming to isolate the constantly increasing difficulty of token generation on long contexts from the challenges of recognizing new token positions. Our experiments on synthetic tasks show that after applying Resonance RoPE, Transformers recognize OOD position better and more robustly. Our extensive LLM experiments also show superior performance after applying Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on both upstream language modeling tasks and a variety of downstream long-text applications.

Title: On Robustness and Generalization of ML-Based Congestion Predictors to Valid and Imperceptible Perturbations

• Authors: Chester Holtz, Yucheng Wang, Chung-Kuan Cheng, Bill Lin
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2403.00103
• Pdf URL: https://arxiv.org/pdf/2403.00103
• Copy Paste: [[2403.00103]] On Robustness and Generalization of ML-Based Congestion Predictors to Valid and Imperceptible Perturbations(https://arxiv.org/abs/2403.00103)
• Keywords: robust
• Abstract: There is substantial interest in the use of machine learning (ML)-based techniques throughout the electronic computer-aided design (CAD) flow, particularly methods based on deep learning. However, while deep learning methods have achieved state-of-the-art performance in several applications, recent work has demonstrated that neural networks are generally vulnerable to small, carefully chosen perturbations of their input (e.g. a single pixel change in an image). In this work, we investigate robustness in the context of ML-based EDA tools -- particularly for congestion prediction. As far as we are aware, we are the first to explore this concept in the context of ML-based EDA. We first describe a novel notion of imperceptibility designed specifically for VLSI layout problems defined on netlists and cell placements. Our definition of imperceptibility is characterized by a guarantee that a perturbation to a layout will not alter its global routing. We then demonstrate that state-of-the-art CNN and GNN-based congestion models exhibit brittleness to imperceptible perturbations. Namely, we show that when a small number of cells (e.g. 1%-5% of cells) have their positions shifted such that a measure of global congestion is guaranteed to remain unaffected (e.g. 1% of the design adversarially shifted by 0.001% of the layout space results in a predicted decrease in congestion of up to 90%, while no change in congestion is implied by the perturbation). In other words, the quality of a predictor can be made arbitrarily poor (i.e. can be made to predict that a design is "congestion-free") for an arbitrary input layout. Next, we describe a simple technique to train predictors that improves robustness to these perturbations. Our work indicates that CAD engineers should be cautious when integrating neural network-based mechanisms in EDA flows to ensure robust and high-quality results.

Title: LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario

• Authors: Hongyi Liu, Zirui Liu, Ruixiang Tang, Jiayi Yuan, Shaochen Zhong, Yu-Neng Chuang, Li Li, Rui Chen, Xia Hu
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00108
• Pdf URL: https://arxiv.org/pdf/2403.00108
• Copy Paste: [[2403.00108]] LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario(https://arxiv.org/abs/2403.00108)
• Keywords: attack
• Abstract: Fine-tuning LLMs is crucial to enhancing their task-specific performance and ensuring model behaviors are aligned with human preferences. Among various fine-tuning methods, LoRA is popular for its efficiency and ease to use, allowing end-users to easily post and adopt lightweight LoRA modules on open-source platforms to tailor their model for different customization. However, such a handy share-and-play setting opens up new attack surfaces, that the attacker can render LoRA as an attacker, such as backdoor injection, and widely distribute the adversarial LoRA to the community easily. This can result in detrimental outcomes. Despite the huge potential risks of sharing LoRA modules, this aspect however has not been fully explored. To fill the gap, in this study we thoroughly investigate the attack opportunities enabled in the growing share-and-play scenario. Specifically, we study how to inject backdoor into the LoRA module and dive deeper into LoRA's infection mechanisms. We found that training-free mechanism is possible in LoRA backdoor injection. We also discover the impact of backdoor attacks with the presence of multiple LoRA adaptions concurrently as well as LoRA based backdoor transferability. Our aim is to raise awareness of the potential risks under the emerging share-and-play scenario, so as to proactively prevent potential consequences caused by LoRA-as-an-Attack. Warning: the paper contains potential offensive content generated by models.

Title: Federated Linear Contextual Bandits with Heterogeneous Clients

• Authors: Ethan Blaser, Chuanhao Li, Hongning Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00116
• Pdf URL: https://arxiv.org/pdf/2403.00116
• Copy Paste: [[2403.00116]] Federated Linear Contextual Bandits with Heterogeneous Clients(https://arxiv.org/abs/2403.00116)
• Keywords: federate
• Abstract: The demand for collaborative and private bandit learning across multiple agents is surging due to the growing quantity of data generated from distributed systems. Federated bandit learning has emerged as a promising framework for private, efficient, and decentralized online learning. However, almost all previous works rely on strong assumptions of client homogeneity, i.e., all participating clients shall share the same bandit model; otherwise, they all would suffer linear regret. This greatly restricts the application of federated bandit learning in practice. In this work, we introduce a new approach for federated bandits for heterogeneous clients, which clusters clients for collaborative bandit learning under the federated learning setting. Our proposed algorithm achieves non-trivial sub-linear regret and communication cost for all clients, subject to the communication protocol under federated learning that at anytime only one model can be shared by the server.

Title: FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition

• Authors: Xiaoqiang Wang, Bang Liu, Lingfei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00126
• Pdf URL: https://arxiv.org/pdf/2403.00126
• Copy Paste: [[2403.00126]] FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition(https://arxiv.org/abs/2403.00126)
• Keywords: large language model
• Abstract: Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. However, such a paradigm fails to comprehensively differentiate the fine-grained language and cognitive skills, rendering the lack of sufficient interpretation to LLMs' capabilities. In this paper, we present FAC$^2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation. Specifically, we formulate LLMs' evaluation in a multi-dimensional and explainable manner by dissociating the language-related capabilities and the cognition-related ones. Besides, through extracting the intermediate reasoning from LLMs, we further break down the process of applying a specific capability into three sub-steps: recalling relevant knowledge, utilizing knowledge, and solving problems. Finally, FAC$^2$E evaluates each sub-step of each fine-grained capability, providing a two-faceted diagnosis for LLMs. Utilizing FAC$^2$E, we identify a common shortfall in knowledge utilization among models and propose a straightforward, knowledge-enhanced method to mitigate this issue. Our results not only showcase promising performance enhancements but also highlight a direction for future LLM advancements.

Title: EROS: Entity-Driven Controlled Policy Document Summarization

• Authors: Joykirat Singh, Sehban Fazili, Rohan Jain, Md Shad Akhtar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00141
• Pdf URL: https://arxiv.org/pdf/2403.00141
• Copy Paste: [[2403.00141]] EROS: Entity-Driven Controlled Policy Document Summarization(https://arxiv.org/abs/2403.00141)
• Keywords: privacy, protect, extraction, interpretability
• Abstract: Privacy policy documents have a crucial role in educating individuals about the collection, usage, and protection of users' personal data by organizations. However, they are notorious for their lengthy, complex, and convoluted language especially involving privacy-related entities. Hence, they pose a significant challenge to users who attempt to comprehend organization's data usage policy. In this paper, we propose to enhance the interpretability and readability of policy documents by using controlled abstractive summarization -- we enforce the generated summaries to include critical privacy-related entities (e.g., data and medium) and organization's rationale (e.g.,target and reason) in collecting those entities. To achieve this, we develop PD-Sum, a policy-document summarization dataset with marked privacy-related entity labels. Our proposed model, EROS, identifies critical entities through a span-based entity extraction model and employs them to control the information content of the summaries using proximal policy optimization (PPO). Comparison shows encouraging improvement over various baselines. Furthermore, we furnish qualitative and human evaluations to establish the efficacy of EROS.

Title: LLMs in Political Science: Heralding a New Era of Visual Analysis

• Authors: Yu Wang, Mengying Xing
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00154
• Pdf URL: https://arxiv.org/pdf/2403.00154
• Copy Paste: [[2403.00154]] LLMs in Political Science: Heralding a New Era of Visual Analysis(https://arxiv.org/abs/2403.00154)
• Keywords: large language model
• Abstract: Interest is increasing among political scientists in leveraging the extensive information available in images. However, the challenge of interpreting these images lies in the need for specialized knowledge in computer vision and access to specialized hardware. As a result, image analysis has been limited to a relatively small group within the political science community. This landscape could potentially change thanks to the rise of large language models (LLMs). This paper aims to raise awareness of the feasibility of using Gemini for image content analysis. A retrospective analysis was conducted on a corpus of 688 images. Content reports were elicited from Gemini for each image and then manually evaluated by the authors. We find that Gemini is highly accurate in performing object detection, which is arguably the most common and fundamental task in image analysis for political scientists. Equally important, we show that it is easy to implement as the entire command consists of a single prompt in natural language; it is fast to run and should meet the time budget of most researchers; and it is free to use and does not require any specialized hardware. In addition, we illustrate how political scientists can leverage Gemini for other image understanding tasks, including face identification, sentiment analysis, and caption generation. Our findings suggest that Gemini and other similar LLMs have the potential to drastically stimulate and accelerate image research in political science and social sciences more broadly.

Title: Privacy-Preserving Distributed Optimization and Learning

• Authors: Ziqin Chen, Yongqiang Wang
• Subjects: cs.LG, cs.CR, cs.GT
• Abstract URL: https://arxiv.org/abs/2403.00157
• Pdf URL: https://arxiv.org/pdf/2403.00157
• Copy Paste: [[2403.00157]] Privacy-Preserving Distributed Optimization and Learning(https://arxiv.org/abs/2403.00157)
• Keywords: privacy, protect
• Abstract: Distributed optimization and learning has recently garnered great attention due to its wide applications in sensor networks, smart grids, machine learning, and so forth. Despite rapid development, existing distributed optimization and learning algorithms require each agent to exchange messages with its neighbors, which may expose sensitive information and raise significant privacy concerns. In this survey paper, we overview privacy-preserving distributed optimization and learning methods. We first discuss cryptography, differential privacy, and other techniques that can be used for privacy preservation and indicate their pros and cons for privacy protection in distributed optimization and learning. We believe that among these approaches, differential privacy is most promising due to its low computational and communication complexities, which are extremely appealing for modern learning based applications with high dimensions of optimization variables. We then introduce several differential-privacy algorithms that can simultaneously ensure privacy and optimization accuracy. Moreover, we provide example applications in several machine learning problems to confirm the real-world effectiveness of these algorithms. Finally, we highlight some challenges in this research domain and discuss future directions.

Title: TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision

• Authors: Yunyi Zhang, Ruozhen Yang, Xueqiang Xu, Jinfeng Xiao, Jiaming Shen, Jiawei Han
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00165
• Pdf URL: https://arxiv.org/pdf/2403.00165
• Copy Paste: [[2403.00165]] TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision(https://arxiv.org/abs/2403.00165)
• Keywords: large language model
• Abstract: Hierarchical text classification aims to categorize each document into a set of classes in a label taxonomy. Most earlier works focus on fully or semi-supervised methods that require a large amount of human annotated data which is costly and time-consuming to acquire. To alleviate human efforts, in this paper, we work on hierarchical text classification with the minimal amount of supervision: using the sole class name of each node as the only supervision. Recently, large language models (LLM) show competitive performance on various tasks through zero-shot prompting, but this method performs poorly in the hierarchical setting, because it is ineffective to include the large and structured label space in a prompt. On the other hand, previous weakly-supervised hierarchical text classification methods only utilize the raw taxonomy skeleton and ignore the rich information hidden in the text corpus that can serve as additional class-indicative features. To tackle the above challenges, we propose TELEClass, Taxonomy Enrichment and LLM-Enhanced weakly-supervised hierarchical text classification, which (1) automatically enriches the label taxonomy with class-indicative topical terms mined from the corpus to facilitate classifier training and (2) utilizes LLMs for both data annotation and creation tailored for the hierarchical label space. Experiments show that TELEClass can outperform previous weakly-supervised hierarchical text classification methods and LLM-based zero-shot prompting methods on two public datasets.

Title: FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything

• Authors: Safouane El Ghazouali, Youssef Mhirit, Ali Oukhrid, Umberto Michelucci, Hichem Nouira
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00175
• Pdf URL: https://arxiv.org/pdf/2403.00175
• Copy Paste: [[2403.00175]] FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything(https://arxiv.org/abs/2403.00175)
• Keywords: robust, extraction, segmentation
• Abstract: In the realm of computer vision, the integration of advanced techniques into the processing of RGB-D camera inputs poses a significant challenge, given the inherent complexities arising from diverse environmental conditions and varying object appearances. Therefore, this paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. Traditional computer vision systems face limitations in simultaneously capturing precise object boundaries and achieving high-precision object detection on depth map as they are mainly proposed for RGB cameras. To address this challenge, FusionVision adopts an integrated approach by merging state-of-the-art object detection techniques, with advanced instance segmentation methods. The integration of these components enables a holistic (unified analysis of information obtained from both color \textit{RGB} and depth \textit{D} channels) interpretation of RGB-D data, facilitating the extraction of comprehensive and accurate object information. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. Subsequently, FastSAM, an innovative semantic segmentation model, is applied to delineate object boundaries, yielding refined segmentation masks. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation, enhancing overall precision in 3D object segmentation. The code and pre-trained models are publicly available at https://github.com/safouaneelg/FusionVision/.

Title: Ask Your Distribution Shift if Pre-Training is Right for You

• Authors: Benjamin Cohen-Wang, Joshua Vendrow, Aleksander Madry
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00194
• Pdf URL: https://arxiv.org/pdf/2403.00194
• Copy Paste: [[2403.00194]] Ask Your Distribution Shift if Pre-Training is Right for You(https://arxiv.org/abs/2403.00194)
• Keywords: robust
• Abstract: Pre-training is a widely used approach to develop models that are robust to distribution shifts. However, in practice, its effectiveness varies: fine-tuning a pre-trained model improves robustness significantly in some cases but not at all in others (compared to training from scratch). In this work, we seek to characterize the failure modes that pre-training can and cannot address. In particular, we focus on two possible failure modes of models under distribution shift: poor extrapolation (e.g., they cannot generalize to a different domain) and biases in the training data (e.g., they rely on spurious features). Our study suggests that, as a rule of thumb, pre-training can help mitigate poor extrapolation but not dataset biases. After providing theoretical motivation and empirical evidence for this finding, we explore two of its implications for developing robust models: (1) pre-training and interventions designed to prevent exploiting biases have complementary robustness benefits, and (2) fine-tuning on a (very) small, non-diverse but de-biased dataset can result in significantly more robust models than fine-tuning on a large and diverse but biased dataset. Code is available at https://github.com/MadryLab/pretraining-distribution-shift-robustness.

Title: Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras

• Authors: Mathias Viborg Andersen, Ross Greer, Andreas Møgelmose, Mohan Trivedi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00196
• Pdf URL: https://arxiv.org/pdf/2403.00196
• Copy Paste: [[2403.00196]] Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras(https://arxiv.org/abs/2403.00196)
• Keywords: generative
• Abstract: Advanced Driver Assistance Systems (ADAS) in intelligent vehicles rely on accurate driver perception within the vehicle cabin, often leveraging a combination of sensing modalities. However, these modalities operate at varying rates, posing challenges for real-time, comprehensive driver state monitoring. This paper addresses the issue of missing data due to sensor frame rate mismatches, introducing a generative model approach to create synthetic yet realistic thermal imagery. We propose using conditional generative adversarial networks (cGANs), specifically comparing the pix2pix and CycleGAN architectures. Experimental results demonstrate that pix2pix outperforms CycleGAN, and utilizing multi-view input styles, especially stacked views, enhances the accuracy of thermal image generation. Moreover, the study evaluates the model's generalizability across different subjects, revealing the importance of individualized training for optimal performance. The findings suggest the potential of generative models in addressing missing frames, advancing driver state monitoring for intelligent vehicles, and underscoring the need for continued research in model generalization and customization.

Title: AXOLOTL: Fairness through Assisted Self-Debiasing of Large Language Model Outputs

• Authors: Sana Ebrahimi, Kaiwen Chen, Abolfazl Asudeh, Gautam Das, Nick Koudas
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00198
• Pdf URL: https://arxiv.org/pdf/2403.00198
• Copy Paste: [[2403.00198]] AXOLOTL: Fairness through Assisted Self-Debiasing of Large Language Model Outputs(https://arxiv.org/abs/2403.00198)
• Keywords: fair, large language model
• Abstract: Pre-trained Large Language Models (LLMs) have significantly advanced natural language processing capabilities but are susceptible to biases present in their training data, leading to unfair outcomes in various applications. While numerous strategies have been proposed to mitigate bias, they often require extensive computational resources and may compromise model performance. In this work, we introduce AXOLOTL, a novel post-processing framework, which operates agnostically across tasks and models, leveraging public APIs to interact with LLMs without direct access to internal parameters. Through a three-step process resembling zero-shot learning, AXOLOTL identifies biases, proposes resolutions, and guides the model to self-debias its outputs. This approach minimizes computational costs and preserves model performance, making AXOLOTL a promising tool for debiasing LLM outputs with broad applicability and ease of use.

Title: Improving Socratic Question Generation using Data Augmentation and Preference Optimization

• Authors: Nischal Ashok Kumar, Andrew Lan
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00199
• Pdf URL: https://arxiv.org/pdf/2403.00199
• Copy Paste: [[2403.00199]] Improving Socratic Question Generation using Data Augmentation and Preference Optimization(https://arxiv.org/abs/2403.00199)
• Keywords: large language model
• Abstract: The Socratic method is a way of guiding students toward solving a problem independently without directly revealing the solution to the problem. Although this method has been shown to significantly improve student learning outcomes, it remains a complex labor-intensive task for instructors. Large language models (LLMs) can be used to augment human effort by automatically generating Socratic questions for students. However, existing methods that involve prompting these LLMs sometimes produce invalid outputs, e.g., those that directly reveal the solution to the problem or provide irrelevant or premature questions. To alleviate this problem, inspired by reinforcement learning with AI feedback (RLAIF), we first propose a data augmentation method to enrich existing Socratic questioning datasets with questions that are invalid in specific ways. Next, we propose a method to optimize open-source LLMs such as LLama 2 to prefer ground-truth questions over generated invalid ones, using direct preference optimization (DPO). Our experiments on a Socratic questions dataset for student code debugging show that a DPO-optimized 7B LLama 2 model can effectively avoid generating invalid questions, and as a result, outperforms existing state-of-the-art prompting methods.

Title: MaskLRF: Self-supervised Pretraining via Masked Autoencoding of Local Reference Frames for Rotation-invariant 3D Point Set Analysis

• Authors: Takahiko Furuya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00206
• Pdf URL: https://arxiv.org/pdf/2403.00206
• Copy Paste: [[2403.00206]] MaskLRF: Self-supervised Pretraining via Masked Autoencoding of Local Reference Frames for Rotation-invariant 3D Point Set Analysis(https://arxiv.org/abs/2403.00206)
• Keywords: segmentation
• Abstract: Following the successes in the fields of vision and language, self-supervised pretraining via masked autoencoding of 3D point set data, or Masked Point Modeling (MPM), has achieved state-of-the-art accuracy in various downstream tasks. However, current MPM methods lack a property essential for 3D point set analysis, namely, invariance against rotation of 3D objects/scenes. Existing MPM methods are thus not necessarily suitable for real-world applications where 3D point sets may have inconsistent orientations. This paper develops, for the first time, a rotation-invariant self-supervised pretraining framework for practical 3D point set analysis. The proposed algorithm, called MaskLRF, learns rotation-invariant and highly generalizable latent features via masked autoencoding of 3D points within Local Reference Frames (LRFs), which are not affected by rotation of 3D point sets. MaskLRF enhances the quality of latent features by integrating feature refinement using relative pose encoding and feature reconstruction using low-level but rich 3D geometry. The efficacy of MaskLRF is validated via extensive experiments on diverse downstream tasks including classification, segmentation, registration, and domain adaptation. I confirm that MaskLRF achieves new state-of-the-art accuracies in analyzing 3D point sets having inconsistent orientations. Code will be available at: https://github.com/takahikof/MaskLRF

Title: Multi-modal Attribute Prompting for Vision-Language Models

• Authors: Xin Liu, Jiamin Wu, Tianzhu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00219
• Pdf URL: https://arxiv.org/pdf/2403.00219
• Copy Paste: [[2403.00219]] Multi-modal Attribute Prompting for Vision-Language Models(https://arxiv.org/abs/2403.00219)
• Keywords: robust
• Abstract: Large pre-trained Vision-Language Models (VLMs), like CLIP, exhibit strong generalization ability to downstream tasks but struggle in few-shot scenarios. Existing prompting techniques primarily focus on global text and image representations, yet overlooking multi-modal attribute characteristics. This limitation hinders the model's ability to perceive fine-grained visual details and restricts its generalization ability to a broader range of unseen classes. To address this issue, we propose a Multi-modal Attribute Prompting method (MAP) by jointly exploring textual attribute prompting, visual attribute prompting, and attribute-level alignment. The proposed MAP enjoys several merits. First, we introduce learnable visual attribute prompts enhanced by textual attribute semantics to adaptively capture visual attributes for images from unknown categories, boosting fine-grained visual perception capabilities for CLIP. Second, the proposed attribute-level alignment complements the global alignment to enhance the robustness of cross-modal alignment for open-vocabulary objects. To our knowledge, this is the first work to establish cross-modal attribute-level alignment for CLIP-based few-shot adaptation. Extensive experimental results on 11 datasets demonstrate that our method performs favorably against state-of-the-art approaches.

Title: Robust Policy Learning via Offline Skill Diffusion

• Authors: Woo Kyung Kim, Minjong Yoo, Honguk Woo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00225
• Pdf URL: https://arxiv.org/pdf/2403.00225
• Copy Paste: [[2403.00225]] Robust Policy Learning via Offline Skill Diffusion(https://arxiv.org/abs/2403.00225)
• Keywords: robust, diffusion
• Abstract: Skill-based reinforcement learning (RL) approaches have shown considerable promise, especially in solving long-horizon tasks via hierarchical structures. These skills, learned task-agnostically from offline datasets, can accelerate the policy learning process for new tasks. Yet, the application of these skills in different domains remains restricted due to their inherent dependency on the datasets, which poses a challenge when attempting to learn a skill-based policy via RL for a target domain different from the datasets' domains. In this paper, we present a novel offline skill learning framework DuSkill which employs a guided Diffusion model to generate versatile skills extended from the limited skills in datasets, thereby enhancing the robustness of policy learning for tasks in different domains. Specifically, we devise a guided diffusion-based skill decoder in conjunction with the hierarchical encoding to disentangle the skill embedding space into two distinct representations, one for encapsulating domain-invariant behaviors and the other for delineating the factors that induce domain variations in the behaviors. Our DuSkill framework enhances the diversity of skills learned offline, thus enabling to accelerate the learning procedure of high-level policies for different domains. Through experiments, we show that DuSkill outperforms other skill-based imitation learning and RL algorithms for several long-horizon tasks, demonstrating its benefits in few-shot imitation and online RL.

Title: OPAF: Optimized Secure Two-Party Computation Protocols for Nonlinear Activation Functions in Recurrent Neural Network

• Authors: Qian Feng, Zhihua Xia, Zhifeng Xu, Jiasi Weng, Jian Weng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.00239
• Pdf URL: https://arxiv.org/pdf/2403.00239
• Copy Paste: [[2403.00239]] OPAF: Optimized Secure Two-Party Computation Protocols for Nonlinear Activation Functions in Recurrent Neural Network(https://arxiv.org/abs/2403.00239)
• Keywords: secure, privacy
• Abstract: Deep neural network (DNN) typically involves convolutions, pooling, and activation function. Due to the growing concern about privacy, privacy-preserving DNN becomes a hot research topic. Generally, the convolution and pooling operations can be supported by additive homomorphic and secure comparison, but the secure implementation of activation functions is not so straightforward for the requirements of accuracy and efficiency, especially for the non-linear ones such as exponential, sigmoid, and tanh functions. This paper pays a special attention to the implementation of such non-linear functions in semi-honest model with two-party settings, for which SIRNN is the current state-of-the-art. Different from previous works, we proposed improved implementations for these functions by using their intrinsic features as well as worthy tiny tricks. At first, we propose a novel and efficient protocol for exponential function by using a divide-and-conquer strategy with most of the computations executed locally. Exponential protocol is widely used in machine learning tasks such as Poisson regression, and is also a key component of sigmoid and tanh functions. Next, we take advantage of the symmetry of sigmoid and Tanh, and fine-tune the inputs to reduce the 2PC building blocks, which helps to save overhead and improve performance. As a result, we implement these functions with fewer fundamental building blocks. The comprehensive evaluations show that our protocols achieve state-of-the-art precision while reducing run-time by approximately 57%, 44%, and 42% for exponential (with only negative inputs), sigmoid, and Tanh functions, respectively.

Title: YOLO-MED : Multi-Task Interaction Network for Biomedical Images

• Authors: Suizhi Huang, Shalayiding Sirejiding, Yuxiang Lu, Yue Ding, Leheng Liu, Hui Zhou, Hongtao Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00245
• Pdf URL: https://arxiv.org/pdf/2403.00245
• Copy Paste: [[2403.00245]] YOLO-MED : Multi-Task Interaction Network for Biomedical Images(https://arxiv.org/abs/2403.00245)
• Keywords: extraction, segmentation
• Abstract: Object detection and semantic segmentation are pivotal components in biomedical image analysis. Current single-task networks exhibit promising outcomes in both detection and segmentation tasks. Multi-task networks have gained prominence due to their capability to simultaneously tackle segmentation and detection tasks, while also accelerating the segmentation inference. Nevertheless, recent multi-task networks confront distinct limitations such as the difficulty in striking a balance between accuracy and inference speed. Additionally, they often overlook the integration of cross-scale features, which is especially important for biomedical image analysis. In this study, we propose an efficient end-to-end multi-task network capable of concurrently performing object detection and semantic segmentation called YOLO-Med. Our model employs a backbone and a neck for multi-scale feature extraction, complemented by the inclusion of two task-specific decoders. A cross-scale task-interaction module is employed in order to facilitate information fusion between various tasks. Our model exhibits promising results in balancing accuracy and speed when evaluated on the Kvasir-seg dataset and a private biomedical image dataset.

Title: Cloud-based Federated Learning Framework for MRI Segmentation

• Authors: Rukesh Prajapati, Amr S. El-Wakeel
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00254
• Pdf URL: https://arxiv.org/pdf/2403.00254
• Copy Paste: [[2403.00254]] Cloud-based Federated Learning Framework for MRI Segmentation(https://arxiv.org/abs/2403.00254)
• Keywords: privacy, federate, segmentation
• Abstract: In contemporary rural healthcare settings, the principal challenge in diagnosing brain images is the scarcity of available data, given that most of the existing deep learning models demand extensive training data to optimize their performance, necessitating centralized processing methods that potentially compromise data privacy. This paper proposes a novel framework tailored for brain tissue segmentation in rural healthcare facilities. The framework employs a deep reinforcement learning (DRL) environment in tandem with a refinement model (RM) deployed locally at rural healthcare sites. The proposed DRL model has a reduced parameter count and practicality for implementation across distributed rural sites. To uphold data privacy and enhance model generalization without transgressing privacy constraints, we employ federated learning (FL) for cooperative model training. We demonstrate the efficacy of our approach by training the network with a limited data set and observing a substantial performance enhancement, mitigating inaccuracies and irregularities in segmentation across diverse sites. Remarkably, the DRL model attains an accuracy of up to 80%, surpassing the capabilities of conventional convolutional neural networks when confronted with data insufficiency. Incorporating our RM results in an additional accuracy improvement of at least 10%, while FL contributes to a further accuracy enhancement of up to 5%. Collectively, the framework achieves an average 92% accuracy rate within rural healthcare settings characterized by data constraints.

Title: Robust deep labeling of radiological emphysema subtypes using squeeze and excitation convolutional neural networks: The MESA Lung and SPIROMICS Studies

• Authors: Artur Wysoczanski, Nabil Ettehadi, Soroush Arabshahi, Yifei Sun, Karen Hinkley Stukovsky, Karol E. Watson, MeiLan K. Han, Erin D Michos, Alejandro P. Comellas, Eric A. Hoffman, Andrew F. Laine, R. Graham Barr, Elsa D. Angelini
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00257
• Pdf URL: https://arxiv.org/pdf/2403.00257
• Copy Paste: [[2403.00257]] Robust deep labeling of radiological emphysema subtypes using squeeze and excitation convolutional neural networks: The MESA Lung and SPIROMICS Studies(https://arxiv.org/abs/2403.00257)
• Keywords: robust, segmentation
• Abstract: Pulmonary emphysema, the progressive, irreversible loss of lung tissue, is conventionally categorized into three subtypes identifiable on pathology and on lung computed tomography (CT) images. Recent work has led to the unsupervised learning of ten spatially-informed lung texture patterns (sLTPs) on lung CT, representing distinct patterns of emphysematous lung parenchyma based on both textural appearance and spatial location within the lung, and which aggregate into 6 robust and reproducible CT Emphysema Subtypes (CTES). Existing methods for sLTP segmentation, however, are slow and highly sensitive to changes in CT acquisition protocol. In this work, we present a robust 3-D squeeze-and-excitation CNN for supervised classification of sLTPs and CTES on lung CT. Our results demonstrate that this model achieves accurate and reproducible sLTP segmentation on lung CTscans, across two independent cohorts and independently of scanner manufacturer and model.

Title: Extracting Polymer Nanocomposite Samples from Full-Length Documents

• Authors: Ghazal Khalighinejad, Defne Circi, L.C. Brinson, Bhuwan Dhingra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00260
• Pdf URL: https://arxiv.org/pdf/2403.00260
• Copy Paste: [[2403.00260]] Extracting Polymer Nanocomposite Samples from Full-Length Documents(https://arxiv.org/abs/2403.00260)
• Keywords: extraction, large language model
• Abstract: This paper investigates the use of large language models (LLMs) for extracting sample lists of polymer nanocomposites (PNCs) from full-length materials science research papers. The challenge lies in the complex nature of PNC samples, which have numerous attributes scattered throughout the text. The complexity of annotating detailed information on PNCs limits the availability of data, making conventional document-level relation extraction techniques impractical due to the challenge in creating comprehensive named entity span annotations. To address this, we introduce a new benchmark and an evaluation technique for this task and explore different prompting strategies in a zero-shot manner. We also incorporate self-consistency to improve the performance. Our findings show that even advanced LLMs struggle to extract all of the samples from an article. Finally, we analyze the errors encountered in this process, categorizing them into three main challenges, and discuss potential strategies for future research to overcome them.

Title: Parameter-Efficient Tuning of Large Convolutional Models

• Authors: Wei Chen, Zichen Miao, Qiang Qiu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00269
• Pdf URL: https://arxiv.org/pdf/2403.00269
• Copy Paste: [[2403.00269]] Parameter-Efficient Tuning of Large Convolutional Models(https://arxiv.org/abs/2403.00269)
• Keywords: diffusion, generative
• Abstract: To address the high computational and parameter complexity associated with fine-tuning large pre-trained models, researchers have developed parameter-efficient methods, where only partial parameters are updated for downstream tasks. However, these works often overlook the distinct properties of convolutional kernels, which still remain essential elements in many large models, such as Stable Diffusion. In this study, we first introduce filter subspace by decomposing convolutional kernels within each network layer over a small set of filter subspace elements, referred to as filter atoms. We then fine-tune these models to extract task-specific representation by only adapting the filter atoms, a few hundred parameters typically. To potentially expand the parameter space for tuning, we further show a simple approach to generate an overcomplete filter subspace by recursively decomposing each filter atom over another set of filter atoms. The fine-tuning of filter atoms reshapes the filter subspace, enabling convolutional layers to adapt to diverse downstream tasks efficiently. Extensive experiments show that such a simple scheme surpasses previous tuning baselines for both discriminate and generative tasks. Our approach can potentially be complementary to many existing fine-tuning methods.

Title: CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation

• Authors: Xi Liu, Ying Guo, Cheng Zhen, Tong Li, Yingying Ao, Pengfei Yan
• Subjects: cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.00274
• Pdf URL: https://arxiv.org/pdf/2403.00274
• Copy Paste: [[2403.00274]] CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation(https://arxiv.org/abs/2403.00274)
• Keywords: diffusion
• Abstract: Listening head generation aims to synthesize a non-verbal responsive listener head by modeling the correlation between the speaker and the listener in dynamic conversion.The applications of listener agent generation in virtual interaction have promoted many works achieving the diverse and fine-grained motion generation. However, they can only manipulate motions through simple emotional labels, but cannot freely control the listener's motions. Since listener agents should have human-like attributes (e.g. identity, personality) which can be freely customized by users, this limits their realism. In this paper, we propose a user-friendly framework called CustomListener to realize the free-form text prior guided listener generation. To achieve speaker-listener coordination, we design a Static to Dynamic Portrait module (SDP), which interacts with speaker information to transform static text into dynamic portrait token with completion rhythm and amplitude information. To achieve coherence between segments, we design a Past Guided Generation Module (PGG) to maintain the consistency of customized listener attributes through the motion prior, and utilize a diffusion-based structure conditioned on the portrait token and the motion prior to realize the controllable generation. To train and evaluate our model, we have constructed two text-annotated listening head datasets based on ViCo and RealTalk, which provide text-video paired labels. Extensive experiments have verified the effectiveness of our model.

Title: Gender Bias in Large Language Models across Multiple Languages

• Authors: Jinman Zhao, Yitian Ding, Chen Jia, Yining Wang, Zifan Qian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00277
• Pdf URL: https://arxiv.org/pdf/2403.00277
• Copy Paste: [[2403.00277]] Gender Bias in Large Language Models across Multiple Languages(https://arxiv.org/abs/2403.00277)
• Keywords: large language model
• Abstract: With the growing deployment of large language models (LLMs) across various applications, assessing the influence of gender biases embedded in LLMs becomes crucial. The topic of gender bias within the realm of natural language processing (NLP) has gained considerable focus, particularly in the context of English. Nonetheless, the investigation of gender bias in languages other than English is still relatively under-explored and insufficiently analyzed. In this work, We examine gender bias in LLMs-generated outputs for different languages. We use three measurements: 1) gender bias in selecting descriptive words given the gender-related context. 2) gender bias in selecting gender-related pronouns (she/he) given the descriptive words. 3) gender bias in the topics of LLM-generated dialogues. We investigate the outputs of the GPT series of LLMs in various languages using our three measurement methods. Our findings revealed significant gender biases across all the languages we examined.

Title: Shifted Interpolation for Differential Privacy

• Authors: Jinho Bok, Weijie Su, Jason M. Altschuler
• Subjects: cs.LG, cs.CR, math.OC, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.00278
• Pdf URL: https://arxiv.org/pdf/2403.00278
• Copy Paste: [[2403.00278]] Shifted Interpolation for Differential Privacy(https://arxiv.org/abs/2403.00278)
• Keywords: privacy
• Abstract: Noisy gradient descent and its variants are the predominant algorithms for differentially private machine learning. It is a fundamental question to quantify their privacy leakage, yet tight characterizations remain open even in the foundational setting of convex losses. This paper improves over previous analyses by establishing (and refining) the "privacy amplification by iteration" phenomenon in the unifying framework of $f$-differential privacy--which tightly captures all aspects of the privacy loss and immediately implies tighter privacy accounting in other notions of differential privacy, e.g., $(\varepsilon,\delta)$-DP and Renyi DP. Our key technical insight is the construction of shifted interpolated processes that unravel the popular shifted-divergences argument, enabling generalizations beyond divergence-based relaxations of DP. Notably, this leads to the first exact privacy analysis in the foundational setting of strongly convex optimization. Our techniques extend to many settings: convex/strongly convex, constrained/unconstrained, full/cyclic/stochastic batches, and all combinations thereof. As an immediate corollary, we recover the $f$-DP characterization of the exponential mechanism for strongly convex optimization in Gopi et al. (2022), and moreover extend this result to more general settings.

Title: SoK: Security of Programmable Logic Controllers

• Authors: Efrén López-Morales (Texas A&M University-Corpus Christi), Ulysse Planta (CISPA Helmholtz Center for Information Security), Carlos Rubio-Medrano (Texas A&M University-Corpus Christi), Ali Abbasi (CISPA Helmholtz Center for Information Security), Alvaro A. Cardenas (University of California, Santa Cruz)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.00280
• Pdf URL: https://arxiv.org/pdf/2403.00280
• Copy Paste: [[2403.00280]] SoK: Security of Programmable Logic Controllers(https://arxiv.org/abs/2403.00280)
• Keywords: security, defense, attack
• Abstract: Billions of people rely on essential utility and manufacturing infrastructures such as water treatment plants, energy management, and food production. Our dependence on reliable infrastructures makes them valuable targets for cyberattacks. One of the prime targets for adversaries attacking physical infrastructures are Programmable Logic Controllers (PLCs) because they connect the cyber and physical worlds. In this study, we conduct the first comprehensive systematization of knowledge that explores the security of PLCs: We present an in-depth analysis of PLC attacks and defenses and discover trends in the security of PLCs from the last 17 years of research. We introduce a novel threat taxonomy for PLCs and Industrial Control Systems (ICS). Finally, we identify and point out research gaps that, if left ignored, could lead to new catastrophic attacks against critical infrastructures.

Title: DPP-Based Adversarial Prompt Searching for Lanugage Models

• Authors: Xu Zhang, Xiaojun Wan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00292
• Pdf URL: https://arxiv.org/pdf/2403.00292
• Copy Paste: [[2403.00292]] DPP-Based Adversarial Prompt Searching for Lanugage Models(https://arxiv.org/abs/2403.00292)
• Keywords: attack
• Abstract: Language models risk generating mindless and offensive content, which hinders their safe deployment. Therefore, it is crucial to discover and modify potential toxic outputs of pre-trained language models before deployment. In this work, we elicit toxic content by automatically searching for a prompt that directs pre-trained language models towards the generation of a specific target output. The problem is challenging due to the discrete nature of textual data and the considerable computational resources required for a single forward pass of the language model. To combat these challenges, we introduce Auto-regressive Selective Replacement Ascent (ASRA), a discrete optimization algorithm that selects prompts based on both quality and similarity with determinantal point process (DPP). Experimental results on six different pre-trained language models demonstrate the efficacy of ASRA for eliciting toxic content. Furthermore, our analysis reveals a strong correlation between the success rate of ASRA attacks and the perplexity of target outputs, while indicating limited association with the quantity of model parameters.

Title: Small, Versatile and Mighty: A Range-View Perception Framework

• Authors: Qiang Meng, Xiao Wang, JiaBao Wang, Liujiang Yan, Ke Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00325
• Pdf URL: https://arxiv.org/pdf/2403.00325
• Copy Paste: [[2403.00325]] Small, Versatile and Mighty: A Range-View Perception Framework(https://arxiv.org/abs/2403.00325)
• Keywords: segmentation
• Abstract: Despite its compactness and information integrity, the range view representation of LiDAR data rarely occurs as the first choice for 3D perception tasks. In this work, we further push the envelop of the range-view representation with a novel multi-task framework, achieving unprecedented 3D detection performances. Our proposed Small, Versatile, and Mighty (SVM) network utilizes a pure convolutional architecture to fully unleash the efficiency and multi-tasking potentials of the range view representation. To boost detection performances, we first propose a range-view specific Perspective Centric Label Assignment (PCLA) strategy, and a novel View Adaptive Regression (VAR) module to further refine hard-to-predict box properties. In addition, our framework seamlessly integrates semantic segmentation and panoptic segmentation tasks for the LiDAR point cloud, without extra modules. Among range-view-based methods, our model achieves new state-of-the-art detection performances on the Waymo Open Dataset. Especially, over 10 mAP improvement over convolutional counterparts can be obtained on the vehicle class. Our presented results for other tasks further reveal the multi-task capabilities of the proposed small but mighty framework.

Title: DAMS-DETR: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion

• Authors: Guo Junjie, Gao Chenqiang, Liu Fangcen, Meng Deyu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00326
• Pdf URL: https://arxiv.org/pdf/2403.00326
• Copy Paste: [[2403.00326]] DAMS-DETR: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion(https://arxiv.org/abs/2403.00326)
• Keywords: robust, transformer
• Abstract: Infrared-visible object detection aims to achieve robust even full-day object detection by fusing the complementary information of infrared and visible images. However, highly dynamically variable complementary characteristics and commonly existing modality misalignment make the fusion of complementary information difficult. In this paper, we propose a Dynamic Adaptive Multispectral Detection Transformer (DAMS-DETR) based on DETR to simultaneously address these two challenges. Specifically, we propose a Modality Competitive Query Selection strategy to provide useful prior information. This strategy can dynamically select basic salient modality feature representation for each object. To effectively mine the complementary information and adapt to misalignment situations, we propose a Multispectral Deformable Cross-attention module to adaptively sample and aggregate multi-semantic level features of infrared and visible images for each object. In addition, we further adopt the cascade structure of DETR to better mine complementary information. Experiments on four public datasets of different scenes demonstrate significant improvements compared to other state-of-the-art methods. The code will be released at https://github.com/gjj45/DAMS-DETR.

Title: Task Indicating Transformer for Task-conditional Dense Predictions

• Authors: Yuxiang Lu, Shalayiding Sirejiding, Bayram Bayramli, Suizhi Huang, Yue Ding, Hongtao Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00327
• Pdf URL: https://arxiv.org/pdf/2403.00327
• Copy Paste: [[2403.00327]] Task Indicating Transformer for Task-conditional Dense Predictions(https://arxiv.org/abs/2403.00327)
• Keywords: transformer
• Abstract: The task-conditional model is a distinctive stream for efficient multi-task learning. Existing works encounter a critical limitation in learning task-agnostic and task-specific representations, primarily due to shortcomings in global context modeling arising from CNN-based architectures, as well as a deficiency in multi-scale feature interaction within the decoder. In this paper, we introduce a novel task-conditional framework called Task Indicating Transformer (TIT) to tackle this challenge. Our approach designs a Mix Task Adapter module within the transformer block, which incorporates a Task Indicating Matrix through matrix decomposition, thereby enhancing long-range dependency modeling and parameter-efficient feature adaptation by capturing intra- and inter-task features. Moreover, we propose a Task Gate Decoder module that harnesses a Task Indicating Vector and gating mechanism to facilitate adaptive multi-scale feature refinement guided by task embeddings. Experiments on two public multi-task dense prediction benchmarks, NYUD-v2 and PASCAL-Context, demonstrate that our approach surpasses state-of-the-art task-conditional methods.

Title: Nonlinear Sheaf Diffusion in Graph Neural Networks

• Authors: Olga Zaghen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00337
• Pdf URL: https://arxiv.org/pdf/2403.00337
• Copy Paste: [[2403.00337]] Nonlinear Sheaf Diffusion in Graph Neural Networks(https://arxiv.org/abs/2403.00337)
• Keywords: diffusion
• Abstract: This work focuses on exploring the potential benefits of introducing a nonlinear Laplacian in Sheaf Neural Networks for graph-related tasks. The primary aim is to understand the impact of such nonlinearity on diffusion dynamics, signal propagation, and performance of neural network architectures in discrete-time settings. The study primarily emphasizes experimental analysis, using real-world and synthetic datasets to validate the practical effectiveness of different versions of the model. This approach shifts the focus from an initial theoretical exploration to demonstrating the practical utility of the proposed model.

Title: Semi-Instruct: Bridging Natural-Instruct and Self-Instruct for Code Large Language Models

• Authors: Xianzhen Luo, Qingfu Zhu, Zhiming Zhang, Xu Wang, Qing Yang, Dongliang Xu, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00338
• Pdf URL: https://arxiv.org/pdf/2403.00338
• Copy Paste: [[2403.00338]] Semi-Instruct: Bridging Natural-Instruct and Self-Instruct for Code Large Language Models(https://arxiv.org/abs/2403.00338)
• Keywords: large language model
• Abstract: Instruction tuning plays a pivotal role in Code Large Language Models (Code LLMs) for the task of program synthesis. Presently, two dominant paradigms for collecting tuning data are natural-instruct (human-written) and self-instruct (automatically generated). Natural-instruct includes diverse and correct codes but lacks instruction-code pairs, and exists improper code formats like nested single-line codes. In contrast, self-instruct automatically generates proper paired data. However, it suffers from low diversity due to generating duplicates and cannot ensure the correctness of codes. To bridge the both paradigms, we propose \textbf{Semi-Instruct}. It first converts diverse but improper codes from natural-instruct into proper instruction-code pairs through a method similar to self-instruct. To verify the correctness of generated codes, we design a novel way to construct test cases by generating cases' inputs and executing correct codes from natural-instruct to get outputs. Finally, diverse and correct instruction-code pairs are retained for instruction tuning. Experiments show that semi-instruct is significantly better than natural-instruct and self-instruct. Furthermore, the performance steadily improves as data scale increases.

Title: Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning

• Authors: Ruiqian Nai, Zixin Wen, Ji Li, Yuanzhi Li, Yang Gao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00352
• Pdf URL: https://arxiv.org/pdf/2403.00352
• Copy Paste: [[2403.00352]] Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning(https://arxiv.org/abs/2403.00352)
• Keywords: generative
• Abstract: In representation learning, a disentangled representation is highly desirable as it encodes generative factors of data in a separable and compact pattern. Researchers have advocated leveraging disentangled representations to complete downstream tasks with encouraging empirical evidence. This paper further investigates the necessity of disentangled representation in downstream applications. Specifically, we show that dimension-wise disentangled representations are unnecessary on a fundamental downstream task, abstract visual reasoning. We provide extensive empirical evidence against the necessity of disentanglement, covering multiple datasets, representation learning methods, and downstream network architectures. Furthermore, our findings suggest that the informativeness of representations is a better indicator of downstream performance than disentanglement. Finally, the positive correlation between informativeness and disentanglement explains the claimed usefulness of disentangled representations in previous works. The source code is available at https://github.com/Richard-coder-Nai/disentanglement-lib-necessity.git.

Title: Self-Consistent Reasoning-based Aspect-Sentiment Quad Prediction with Extract-Then-Assign Strategy

• Authors: Jieyong Kim, Ryang Heo, Yongsik Seo, SeongKu Kang, Jinyoung Yeo, Dongha Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00354
• Pdf URL: https://arxiv.org/pdf/2403.00354
• Copy Paste: [[2403.00354]] Self-Consistent Reasoning-based Aspect-Sentiment Quad Prediction with Extract-Then-Assign Strategy(https://arxiv.org/abs/2403.00354)
• Keywords: interpretability, generative
• Abstract: In the task of aspect sentiment quad prediction (ASQP), generative methods for predicting sentiment quads have shown promising results. However, they still suffer from imprecise predictions and limited interpretability, caused by data scarcity and inadequate modeling of the quadruplet composition process. In this paper, we propose Self-Consistent Reasoning-based Aspect-sentiment quadruple Prediction (SCRAP), optimizing its model to generate reasonings and the corresponding sentiment quadruplets in sequence. SCRAP adopts the Extract-Then-Assign reasoning strategy, which closely mimics human cognition. In the end, SCRAP significantly improves the model's ability to handle complex reasoning tasks and correctly predict quadruplets through consistency voting, resulting in enhanced interpretability and accuracy in ASQP.

Title: HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generation

• Authors: Zhiying Leng, Tolga Birdal, Xiaohui Liang, Federico Tombari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00372
• Pdf URL: https://arxiv.org/pdf/2403.00372
• Copy Paste: [[2403.00372]] HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generation(https://arxiv.org/abs/2403.00372)
• Keywords: diffusion
• Abstract: 3D shape generation from text is a fundamental task in 3D representation learning. The text-shape pairs exhibit a hierarchical structure, where a general text like "chair" covers all 3D shapes of the chair, while more detailed prompts refer to more specific shapes. Furthermore, both text and 3D shapes are inherently hierarchical structures. However, existing Text2Shape methods, such as SDFusion, do not exploit that. In this work, we propose HyperSDFusion, a dual-branch diffusion model that generates 3D shapes from a given text. Since hyperbolic space is suitable for handling hierarchical data, we propose to learn the hierarchical representations of text and 3D shapes in hyperbolic space. First, we introduce a hyperbolic text-image encoder to learn the sequential and multi-modal hierarchical features of text in hyperbolic space. In addition, we design a hyperbolic text-graph convolution module to learn the hierarchical features of text in hyperbolic space. In order to fully utilize these text features, we introduce a dual-branch structure to embed text features in 3D feature space. At last, to endow the generated 3D shapes with a hierarchical structure, we devise a hyperbolic hierarchical loss. Our method is the first to explore the hyperbolic hierarchical representation for text-to-shape generation. Experimental results on the existing text-to-shape paired dataset, Text2Shape, achieved state-of-the-art results.

Title: GLFNET: Global-Local (frequency) Filter Networks for efficient medical image segmentation

• Authors: Athanasios Tragakis, Qianying Liu, Chaitanya Kaul, Swalpa Kumar Roy, Hang Dai, Fani Deligianni, Roderick Murray-Smith, Daniele Faccio
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00396
• Pdf URL: https://arxiv.org/pdf/2403.00396
• Copy Paste: [[2403.00396]] GLFNET: Global-Local (frequency) Filter Networks for efficient medical image segmentation(https://arxiv.org/abs/2403.00396)
• Keywords: extraction, transformer, segmentation
• Abstract: We propose a novel transformer-style architecture called Global-Local Filter Network (GLFNet) for medical image segmentation and demonstrate its state-of-the-art performance. We replace the self-attention mechanism with a combination of global-local filter blocks to optimize model efficiency. The global filters extract features from the whole feature map whereas the local filters are being adaptively created as 4x4 patches of the same feature map and add restricted scale information. In particular, the feature extraction takes place in the frequency domain rather than the commonly used spatial (image) domain to facilitate faster computations. The fusion of information from both spatial and frequency spaces creates an efficient model with regards to complexity, required data and performance. We test GLFNet on three benchmark datasets achieving state-of-the-art performance on all of them while being almost twice as efficient in terms of GFLOP operations.

Title: Secure Routing for Mobile Ad hoc Networks

• Authors: Panagiotis Papadimitratos, Zygmunt J. Haas
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.00404
• Pdf URL: https://arxiv.org/pdf/2403.00404
• Copy Paste: [[2403.00404]] Secure Routing for Mobile Ad hoc Networks(https://arxiv.org/abs/2403.00404)
• Keywords: secure, security, attack, robust
• Abstract: The emergence of the Mobile Ad Hoc Networking (MANET) technology advocates self-organized wireless interconnection of communication devices that would either extend or operate in concert with the wired networking infrastructure or, possibly, evolve to autonomous networks. In either case, the proliferation of MANET-based applications depends on a multitude of factors, with trustworthiness being one of the primary challenges to be met. Despite the existence of well-known security mechanisms, additional vulnerabilities and features pertinent to this new networking paradigm might render such traditional solutions inapplicable. In particular, the absence of a central authorization facility in an open and distributed communication environment is a major challenge, especially due to the need for cooperative network operation. In particular, in MANET, any node may compromise the routing protocol functionality by disrupting the route discovery process. In this paper, we present a route discovery protocol that mitigates the detrimental effects of such malicious behavior, as to provide correct connectivity information. Our protocol guarantees that fabricated, compromised, or replayed route replies would either be rejected or never reach back the querying node. Furthermore, the protocol responsiveness is safeguarded under different types of attacks that exploit the routing protocol itself. The sole requirement of the proposed scheme is the existence of a security association between the node initiating the query and the sought destination. Specifically, no assumption is made regarding the intermediate nodes, which may exhibit arbitrary and malicious behavior. The scheme is robust in the presence of a number of non-colluding nodes, and provides accurate routing information in a timely manner.

Title: SoK: Cross-Chain Bridging Architectural Design Flaws and Mitigations

• Authors: Jakob Svennevik Notland, Jinguye Li, Mariusz Nowostawski, Peter Halland Haro
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.00405
• Pdf URL: https://arxiv.org/pdf/2403.00405
• Copy Paste: [[2403.00405]] SoK: Cross-Chain Bridging Architectural Design Flaws and Mitigations(https://arxiv.org/abs/2403.00405)
• Keywords: secure, security
• Abstract: Cross-chain bridges are solutions that enable interoperability between heterogeneous blockchains. In contrast to the underlying blockchains, the bridges often provide inferior security guarantees and have been targets of hacks causing damage in the range of 1.5 to 2 billion USD in 2022. The current state of bridge architectures is that they are ambiguous, and there is next to no notion of how different architectures and their components are related to different vulnerabilities. Throughout this study, we have analysed 60 different bridges and 34 bridge exploits in the last three years (2021-2023). Our analyses identified 13 architectural components of the bridges. We linked the components to eight types of vulnerabilities, also called design flaws. We identified prevention measures and proposed 11 impact reduction measures based on the existing and possible countermeasures to address the imminent exploits of the design flaws. The results are meant to be used as guidelines for designing and implementing secure cross-chain bridge architectures, preventing design flaws, and mitigating the negative impacts of exploits.

Title: Adaptive Restructuring of Merkle and Verkle Trees for Enhanced Blockchain Scalability

• Authors: Oleksandr Kuznetsov, Dzianis Kanonik, Alex Rusnak, Anton Yezhov, Oleksandr Domin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.00406
• Pdf URL: https://arxiv.org/pdf/2403.00406
• Copy Paste: [[2403.00406]] Adaptive Restructuring of Merkle and Verkle Trees for Enhanced Blockchain Scalability(https://arxiv.org/abs/2403.00406)
• Keywords: secure, security
• Abstract: The scalability of blockchain technology remains a pivotal challenge, impeding its widespread adoption across various sectors. This study introduces an innovative approach to address this challenge by proposing the adaptive restructuring of Merkle and Verkle trees, fundamental components of blockchain architecture responsible for ensuring data integrity and facilitating efficient verification processes. Unlike traditional static tree structures, our adaptive model dynamically adjusts the configuration of these trees based on usage patterns, significantly reducing the average path length required for verification and, consequently, the computational overhead associated with these processes. Through a comprehensive conceptual framework, we delineate the methodology for adaptive restructuring, encompassing both binary and non-binary tree configurations. This framework is validated through a series of detailed examples, demonstrating the practical feasibility and the efficiency gains achievable with our approach. Moreover, we present a comparative analysis with existing scalability solutions, highlighting the unique advantages of adaptive restructuring in terms of simplicity, security, and efficiency enhancement without introducing additional complexities or dependencies. This study's implications extend beyond theoretical advancements, offering a scalable, secure, and efficient method for blockchain data verification that could facilitate broader adoption of blockchain technology in finance, supply chain management, and beyond. As the blockchain ecosystem continues to evolve, the principles and methodologies outlined herein are poised to contribute significantly to its growth and maturity.

Title: Provably Robust DPO: Aligning Language Models with Noisy Feedback

• Authors: Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00409
• Pdf URL: https://arxiv.org/pdf/2403.00409
• Copy Paste: [[2403.00409]] Provably Robust DPO: Aligning Language Models with Noisy Feedback(https://arxiv.org/abs/2403.00409)
• Keywords: robust, generative
• Abstract: Learning from preference-based feedback has recently gained traction as a promising approach to align language models with human interests. While these aligned generative models have demonstrated impressive capabilities across various tasks, their dependence on high-quality human preference data poses a bottleneck in practical applications. Specifically, noisy (incorrect and ambiguous) preference pairs in the dataset might restrict the language models from capturing human intent accurately. While practitioners have recently proposed heuristics to mitigate the effect of noisy preferences, a complete theoretical understanding of their workings remain elusive. In this work, we aim to bridge this gap by by introducing a general framework for policy optimization in the presence of random preference flips. We focus on the direct preference optimization (DPO) algorithm in particular since it assumes that preferences adhere to the Bradley-Terry-Luce (BTL) model, raising concerns about the impact of noisy data on the learned policy. We design a novel loss function, which de-bias the effect of noise on average, making a policy trained by minimizing that loss robust to the noise. Under log-linear parameterization of the policy class and assuming good feature coverage of the SFT policy, we prove that the sub-optimality gap of the proposed robust DPO (rDPO) policy compared to the optimal policy is of the order $O(\frac{1}{1-2\epsilon}\sqrt{\frac{d}{n}})$, where $\epsilon < 1/2$ is flip rate of labels, $d$ is policy parameter dimension and $n$ is size of dataset. Our experiments on IMDb sentiment generation and Anthropic's helpful-harmless dataset show that rDPO is robust to noise in preference labels compared to vanilla DPO and other heuristics proposed by practitioners.

Title: Cross-Lingual Learning vs. Low-Resource Fine-Tuning: A Case Study with Fact-Checking in Turkish

• Authors: Recep Firat Cekinel, Pinar Karagoz, Cagri Coltekin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00411
• Pdf URL: https://arxiv.org/pdf/2403.00411
• Copy Paste: [[2403.00411]] Cross-Lingual Learning vs. Low-Resource Fine-Tuning: A Case Study with Fact-Checking in Turkish(https://arxiv.org/abs/2403.00411)
• Keywords: large language model
• Abstract: The rapid spread of misinformation through social media platforms has raised concerns regarding its impact on public opinion. While misinformation is prevalent in other languages, the majority of research in this field has concentrated on the English language. Hence, there is a scarcity of datasets for other languages, including Turkish. To address this concern, we have introduced the FCTR dataset, consisting of 3238 real-world claims. This dataset spans multiple domains and incorporates evidence collected from three Turkish fact-checking organizations. Additionally, we aim to assess the effectiveness of cross-lingual transfer learning for low-resource languages, with a particular focus on Turkish. We demonstrate in-context learning (zero-shot and few-shot) performance of large language models in this context. The experimental results indicate that the dataset has the potential to advance research in the Turkish language.

Title: Robust Deep Reinforcement Learning Through Adversarial Attacks and Training : A Survey

• Authors: Lucas Schott, Josephine Delas, Hatem Hajri, Elies Gherbi, Reda Yaich, Nora Boulahia-Cuppens, Frederic Cuppens, Sylvain Lamprier
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00420
• Pdf URL: https://arxiv.org/pdf/2403.00420
• Copy Paste: [[2403.00420]] Robust Deep Reinforcement Learning Through Adversarial Attacks and Training : A Survey(https://arxiv.org/abs/2403.00420)
• Keywords: attack, robust
• Abstract: Deep Reinforcement Learning (DRL) is an approach for training autonomous agents across various complex environments. Despite its significant performance in well known environments, it remains susceptible to minor conditions variations, raising concerns about its reliability in real-world applications. To improve usability, DRL must demonstrate trustworthiness and robustness. A way to improve robustness of DRL to unknown changes in the conditions is through Adversarial Training, by training the agent against well suited adversarial attacks on the dynamics of the environment. Addressing this critical issue, our work presents an in-depth analysis of contemporary adversarial attack methodologies, systematically categorizing them and comparing their objectives and operational mechanisms. This classification offers a detailed insight into how adversarial attacks effectively act for evaluating the resilience of DRL agents, thereby paving the way for enhancing their robustness.

Title: HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding

• Authors: Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, Jiawei Zhou
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00425
• Pdf URL: https://arxiv.org/pdf/2403.00425
• Copy Paste: [[2403.00425]] HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding(https://arxiv.org/abs/2403.00425)
• Keywords: robust
• Abstract: While large vision-language models (LVLMs) have demonstrated impressive capabilities in interpreting multi-modal contexts, they invariably suffer from object hallucinations (OH). We introduce HALC, a novel decoding algorithm designed to mitigate OH in LVLMs. HALC leverages distinct fine-grained optimal visual information in vision-language tasks and operates on both local and global contexts simultaneously. Specifically, HALC integrates a robust auto-focal grounding mechanism (locally) to correct hallucinated tokens on the fly, and a specialized beam search algorithm (globally) to significantly reduce OH while preserving text generation quality. Additionally, HALC can be integrated into any LVLMs as a plug-and-play module without extra training. Extensive experimental studies demonstrate the effectiveness of HALC in reducing OH, outperforming state-of-the-arts across four benchmarks.

Title: Hierarchical Indexing for Retrieval-Augmented Opinion Summarization

• Authors: Tom Hosking, Hao Tang, Mirella Lapata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00435
• Pdf URL: https://arxiv.org/pdf/2403.00435
• Copy Paste: [[2403.00435]] Hierarchical Indexing for Retrieval-Augmented Opinion Summarization(https://arxiv.org/abs/2403.00435)
• Keywords: large language model
• Abstract: We propose a method for unsupervised abstractive opinion summarization, that combines the attributability and scalability of extractive approaches with the coherence and fluency of Large Language Models (LLMs). Our method, HIRO, learns an index structure that maps sentences to a path through a semantically organized discrete hierarchy. At inference time, we populate the index and use it to identify and retrieve clusters of sentences containing popular opinions from input reviews. Then, we use a pretrained LLM to generate a readable summary that is grounded in these extracted evidential clusters. The modularity of our approach allows us to evaluate its efficacy at each stage. We show that HIRO learns an encoding space that is more semantically structured than prior work, and generates summaries that are more representative of the opinions in the input reviews. Human evaluation confirms that HIRO generates more coherent, detailed and accurate summaries that are significantly preferred by annotators compared to prior work.

Title: Abductive Ego-View Accident Video Understanding for Safe Driving Perception

• Authors: Jianwu Fang, Lei-lei Li, Junfei Zhou, Junbin Xiao, Hongkai Yu, Chen Lv, Jianru Xue, Tat-Seng Chua
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00436
• Pdf URL: https://arxiv.org/pdf/2403.00436
• Copy Paste: [[2403.00436]] Abductive Ego-View Accident Video Understanding for Safe Driving Perception(https://arxiv.org/abs/2403.00436)
• Keywords: diffusion
• Abstract: We present MM-AU, a novel dataset for Multi-Modal Accident video Understanding. MM-AU contains 11,727 in-the-wild ego-view accident videos, each with temporally aligned text descriptions. We annotate over 2.23 million object boxes and 58,650 pairs of video-based accident reasons, covering 58 accident categories. MM-AU supports various accident understanding tasks, particularly multimodal video diffusion to understand accident cause-effect chains for safe driving. With MM-AU, we present an Abductive accident Video understanding framework for Safe Driving perception (AdVersa-SD). AdVersa-SD performs video diffusion via an Object-Centric Video Diffusion (OAVD) method which is driven by an abductive CLIP model. This model involves a contrastive interaction loss to learn the pair co-occurrence of normal, near-accident, accident frames with the corresponding text descriptions, such as accident reasons, prevention advice, and accident categories. OAVD enforces the causal region learning while fixing the content of the original frame background in video generation, to find the dominant cause-effect chain for certain accidents. Extensive experiments verify the abductive ability of AdVersa-SD and the superiority of OAVD against the state-of-the-art diffusion models. Additionally, we provide careful benchmark evaluations for object detection and accident reason answering since AdVersa-SD relies on precise object and accident reason information.

Title: LoMOE: Localized Multi-Object Editing via Multi-Diffusion

• Authors: Goirik Chakrabarty, Aditya Chandrasekar, Ramya Hebbalaguppe, Prathosh AP
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00437
• Pdf URL: https://arxiv.org/pdf/2403.00437
• Copy Paste: [[2403.00437]] LoMOE: Localized Multi-Object Editing via Multi-Diffusion(https://arxiv.org/abs/2403.00437)
• Keywords: diffusion
• Abstract: Recent developments in the field of diffusion models have demonstrated an exceptional capacity to generate high-quality prompt-conditioned image edits. Nevertheless, previous approaches have primarily relied on textual prompts for image editing, which tend to be less effective when making precise edits to specific objects or fine-grained regions within a scene containing single/multiple objects. We introduce a novel framework for zero-shot localized multi-object editing through a multi-diffusion process to overcome this challenge. This framework empowers users to perform various operations on objects within an image, such as adding, replacing, or editing $\textbf{many}$ objects in a complex scene $\textbf{in one pass}$. Our approach leverages foreground masks and corresponding simple text prompts that exert localized influences on the target regions resulting in high-fidelity image editing. A combination of cross-attention and background preservation losses within the latent space ensures that the characteristics of the object being edited are preserved while simultaneously achieving a high-quality, seamless reconstruction of the background with fewer artifacts compared to the current methods. We also curate and release a dataset dedicated to multi-object editing, named $\texttt{LoMOE}$-Bench. Our experiments against existing state-of-the-art methods demonstrate the improved effectiveness of our approach in terms of both image editing quality and inference speed.

Title: An Ordinal Diffusion Model for Generating Medical Images with Different Severity Levels

• Authors: Shumpei Takezaki, Seiichi Uchida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00452
• Pdf URL: https://arxiv.org/pdf/2403.00452
• Copy Paste: [[2403.00452]] An Ordinal Diffusion Model for Generating Medical Images with Different Severity Levels(https://arxiv.org/abs/2403.00452)
• Keywords: diffusion, generative
• Abstract: Diffusion models have recently been used for medical image generation because of their high image quality. In this study, we focus on generating medical images with ordinal classes, which have ordinal relationships, such as severity levels. We propose an Ordinal Diffusion Model (ODM) that controls the ordinal relationships of the estimated noise images among the classes. Our model was evaluated experimentally by generating retinal and endoscopic images of multiple severity classes. ODM achieved higher performance than conventional generative models by generating realistic images, especially in high-severity classes with fewer training samples.

Title: Deformable One-shot Face Stylization via DINO Semantic Guidance

• Authors: Yang Zhou, Zichong Chen, Hui Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00459
• Pdf URL: https://arxiv.org/pdf/2403.00459
• Copy Paste: [[2403.00459]] Deformable One-shot Face Stylization via DINO Semantic Guidance(https://arxiv.org/abs/2403.00459)
• Keywords: robust, transformer
• Abstract: This paper addresses the complex issue of one-shot face stylization, focusing on the simultaneous consideration of appearance and structure, where previous methods have fallen short. We explore deformation-aware face stylization that diverges from traditional single-image style reference, opting for a real-style image pair instead. The cornerstone of our method is the utilization of a self-supervised vision transformer, specifically DINO-ViT, to establish a robust and consistent facial structure representation across both real and style domains. Our stylization process begins by adapting the StyleGAN generator to be deformation-aware through the integration of spatial transformers (STN). We then introduce two innovative constraints for generator fine-tuning under the guidance of DINO semantics: i) a directional deformation loss that regulates directional vectors in DINO space, and ii) a relative structural consistency constraint based on DINO token self-similarities, ensuring diverse generation. Additionally, style-mixing is employed to align the color generation with the reference, minimizing inconsistent correspondences. This framework delivers enhanced deformability for general one-shot face stylization, achieving notable efficiency with a fine-tuning duration of approximately 10 minutes. Extensive qualitative and quantitative comparisons demonstrate our superiority over state-of-the-art one-shot face stylization methods. Code is available at \url{https://github.com/zichongc/DoesFS}.

Title: LUCID: LLM-Generated Utterances for Complex and Interesting Dialogues

• Authors: Joe Stacey, Jianpeng Cheng, John Torr, Tristan Guigue, Joris Driesen, Alexandru Coca, Mark Gaynor, Anders Johannsen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00462
• Pdf URL: https://arxiv.org/pdf/2403.00462
• Copy Paste: [[2403.00462]] LUCID: LLM-Generated Utterances for Complex and Interesting Dialogues(https://arxiv.org/abs/2403.00462)
• Keywords: transformer, large language model
• Abstract: Virtual assistants are poised to take a dramatic leap forward in terms of their dialogue capabilities, spurred by recent advances in transformer-based Large Language Models (LLMs). Yet a major bottleneck to achieving genuinely transformative task-oriented dialogue capabilities remains the scarcity of high quality and linguistically sophisticated data. Existing datasets, while impressive in scale, have limited domain coverage and contain few genuinely challenging conversational phenomena; those which are present are typically unlabelled, making it difficult to assess the strengths and weaknesses of models without time-consuming and costly human evaluation. Moreover, creating high quality dialogue data has until now required considerable human input, limiting both the scale of these datasets and the ability to rapidly bootstrap data for a new target domain. We aim to overcome these issues with LUCID, a modularised and highly automated LLM-driven data generation system that produces realistic, diverse and challenging dialogues. We use LUCID to generate a seed dataset of 4,277 multi-domain, multi-intent conversations across 100 intents to demonstrate its capabilities. The generated conversations include a wide range of challenging phenomena and diverse user behaviour, conveniently identifiable via a set of turn-level tags. Finally, we provide separate test sets for seen and unseen intents, allowing for convenient out-of-distribution evaluation. We release both the data generation code and the dataset itself.

Title: Attacking Delay-based PUFs with Minimal Adversary Model

• Authors: Hongming Fei, Owen Millwood, Prosanta Gope, Jack Miskelly, Biplab Sikdar
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2403.00464
• Pdf URL: https://arxiv.org/pdf/2403.00464
• Copy Paste: [[2403.00464]] Attacking Delay-based PUFs with Minimal Adversary Model(https://arxiv.org/abs/2403.00464)
• Keywords: secure, security, attack, fair
• Abstract: Physically Unclonable Functions (PUFs) provide a streamlined solution for lightweight device authentication. Delay-based Arbiter PUFs, with their ease of implementation and vast challenge space, have received significant attention; however, they are not immune to modelling attacks that exploit correlations between their inputs and outputs. Research is therefore polarized between developing modelling-resistant PUFs and devising machine learning attacks against them. This dichotomy often results in exaggerated concerns and overconfidence in PUF security, primarily because there lacks a universal tool to gauge a PUF's security. In many scenarios, attacks require additional information, such as PUF type or configuration parameters. Alarmingly, new PUFs are often branded `secure' if they lack a specific attack model upon introduction. To impartially assess the security of delay-based PUFs, we present a generic framework featuring a Mixture-of-PUF-Experts (MoPE) structure for mounting attacks on various PUFs with minimal adversarial knowledge, which provides a way to compare their performance fairly and impartially. We demonstrate the capability of our model to attack different PUF types, including the first successful attack on Heterogeneous Feed-Forward PUFs using only a reasonable amount of challenges and responses. We propose an extension version of our model, a Multi-gate Mixture-of-PUF-Experts (MMoPE) structure, facilitating multi-task learning across diverse PUFs to recognise commonalities across PUF designs. This allows a streamlining of training periods for attacking multiple PUFs simultaneously. We conclude by showcasing the potent performance of MoPE and MMoPE across a spectrum of PUF types, employing simulated, real-world unbiased, and biased data sets for analysis.

Title: When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability

• Authors: Wenjie Xuan, Yufei Xu, Shanshan Zhao, Chaoyue Wang, Juhua Liu, Bo Du, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00467
• Pdf URL: https://arxiv.org/pdf/2403.00467
• Copy Paste: [[2403.00467]] When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability(https://arxiv.org/abs/2403.00467)
• Keywords: robust
• Abstract: ControlNet excels at creating content that closely matches precise contours in user-provided masks. However, when these masks contain noise, as a frequent occurrence with non-expert users, the output would include unwanted artifacts. This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. Subsequently, to enhance controllability with inexplicit masks, an advanced Shape-aware ControlNet consisting of a deterioration estimator and a shape-prior modulation block is devised. The deterioration estimator assesses the deterioration factor of the provided masks. Then this factor is utilized in the modulation block to adaptively modulate the model's contour-following ability, which helps it dismiss the noise part in the inexplicit masks. Extensive experiments prove its effectiveness in encouraging ControlNet to interpret inaccurate spatial conditions robustly rather than blindly following the given contours. We showcase application scenarios like modifying shape priors and composable shape-controllable generation. Codes are soon available.

Title: TempCompass: Do Video LLMs Really Understand Videos?

• Authors: Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, Lu Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00476
• Pdf URL: https://arxiv.org/pdf/2403.00476
• Copy Paste: [[2403.00476]] TempCompass: Do Video LLMs Really Understand Videos?(https://arxiv.org/abs/2403.00476)
• Keywords: large language model
• Abstract: Recently, there is a surge in interest surrounding video large language models (Video LLMs). However, existing benchmarks fail to provide a comprehensive feedback on the temporal perception ability of Video LLMs. On the one hand, most of them are unable to distinguish between different temporal aspects (e.g., speed, direction) and thus cannot reflect the nuanced performance on these specific aspects. On the other hand, they are limited in the diversity of task formats (e.g., only multi-choice QA), which hinders the understanding of how temporal perception performance may vary across different types of tasks. Motivated by these two problems, we propose the \textbf{TempCompass} benchmark, which introduces a diversity of temporal aspects and task formats. To collect high-quality test data, we devise two novel strategies: (1) In video collection, we construct conflicting videos that share the same static content but differ in a specific temporal aspect, which prevents Video LLMs from leveraging single-frame bias or language priors. (2) To collect the task instructions, we propose a paradigm where humans first annotate meta-information for a video and then an LLM generates the instruction. We also design an LLM-based approach to automatically and accurately evaluate the responses from Video LLMs. Based on TempCompass, we comprehensively evaluate 8 state-of-the-art (SOTA) Video LLMs and 3 Image LLMs, and reveal the discerning fact that these models exhibit notably poor temporal perception ability. Our data will be available at \url{https://github.com/llyx97/TempCompass}.

Title: Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese

• Authors: Yuqi Chen, Sixuan Li, Ying Li, Mohammad Atari
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.00509
• Pdf URL: https://arxiv.org/pdf/2403.00509
• Copy Paste: [[2403.00509]] Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese(https://arxiv.org/abs/2403.00509)
• Keywords: transformer
• Abstract: In this work, we develop a pipeline for historical-psychological text analysis in classical Chinese. Humans have produced texts in various languages for thousands of years; however, most of the computational literature is focused on contemporary languages and corpora. The emerging field of historical psychology relies on computational techniques to extract aspects of psychology from historical corpora using new methods developed in natural language processing (NLP). The present pipeline, called Contextualized Construct Representations (CCR), combines expert knowledge in psychometrics (i.e., psychological surveys) with text representations generated via transformer-based language models to measure psychological constructs such as traditionalism, norm strength, and collectivism in classical Chinese corpora. Considering the scarcity of available data, we propose an indirect supervised contrastive learning approach and build the first Chinese historical psychology corpus (C-HI-PSY) to fine-tune pre-trained models. We evaluate the pipeline to demonstrate its superior performance compared with other approaches. The CCR method outperforms word-embedding-based approaches across all of our tasks and exceeds prompting with GPT-4 in most tasks. Finally, we benchmark the pipeline against objective, external data to further verify its validity.

Title: ROME: Memorization Insights from Text, Probability and Hidden State in Large Language Models

• Authors: Bo Li, Qinghua Zhao, Lijie Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00510
• Pdf URL: https://arxiv.org/pdf/2403.00510
• Copy Paste: [[2403.00510]] ROME: Memorization Insights from Text, Probability and Hidden State in Large Language Models(https://arxiv.org/abs/2403.00510)
• Keywords: large language model
• Abstract: Probing the memorization of large language models holds significant importance. Previous works have established metrics for quantifying memorization, explored various influencing factors, such as data duplication, model size, and prompt length, and evaluated memorization by comparing model outputs with training corpora. However, the training corpora are of enormous scale and its pre-processing is time-consuming. To explore memorization without accessing training data, we propose a novel approach, named ROME, wherein memorization is explored by comparing disparities across memorized and non-memorized. Specifically, models firstly categorize the selected samples into memorized and non-memorized groups, and then comparing the demonstrations in the two groups from the insights of text, probability, and hidden state. Experimental findings show the disparities in factors including word length, part-of-speech, word frequency, mean and variance, just to name a few.

Title: Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning

• Authors: Michal Nauman, Michał Bortkiewicz, Mateusz Ostaszewski, Piotr Miłoś, Tomasz Trzciński, Marek Cygan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00514
• Pdf URL: https://arxiv.org/pdf/2403.00514
• Copy Paste: [[2403.00514]] Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning(https://arxiv.org/abs/2403.00514)
• Keywords: robust
• Abstract: Recent advancements in off-policy Reinforcement Learning (RL) have significantly improved sample efficiency, primarily due to the incorporation of various forms of regularization that enable more gradient update steps than traditional agents. However, many of these techniques have been tested in limited settings, often on tasks from single simulation benchmarks and against well-known algorithms rather than a range of regularization approaches. This limits our understanding of the specific mechanisms driving RL improvements. To address this, we implemented over 60 different off-policy agents, each integrating established regularization techniques from recent state-of-the-art algorithms. We tested these agents across 14 diverse tasks from 2 simulation benchmarks. Our findings reveal that while the effectiveness of a specific regularization setup varies with the task, certain combinations consistently demonstrate robust and superior performance. Notably, a simple Soft Actor-Critic agent, appropriately regularized, reliably solves dog tasks, which were previously solved mainly through model-based approaches.

Title: VisionLLaMA: A Unified LLaMA Interface for Vision Tasks

• Authors: Xiangxiang Chu, Jianlin Su, Bo Zhang, Chunhua Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00522
• Pdf URL: https://arxiv.org/pdf/2403.00522
• Copy Paste: [[2403.00522]] VisionLLaMA: A Unified LLaMA Interface for Vision Tasks(https://arxiv.org/abs/2403.00522)
• Keywords: transformer, large language model
• Abstract: Large language models are built on top of a transformer-based architecture to process textual inputs. For example, the LLaMA stands out among many open-source implementations. Can the same transformer be used to process 2D images? In this paper, we answer this question by unveiling a LLaMA-like vision transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored for this purpose. VisionLLaMA is a unified and generic modelling framework for solving most vision tasks. We extensively evaluate its effectiveness using typical pre-training paradigms in a good portion of downstream tasks of image perception and especially image generation. In many cases, VisionLLaMA have exhibited substantial gains over the previous state-of-the-art vision transformers. We believe that VisionLLaMA can serve as a strong new baseline model for vision generation and understanding. Our code will be released at https://github.com/Meituan-AutoML/VisionLLaMA.

Title: Large Language Models for Simultaneous Named Entity Extraction and Spelling Correction

• Authors: Edward Whittaker, Ikuo Kitagishi
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00528
• Pdf URL: https://arxiv.org/pdf/2403.00528
• Copy Paste: [[2403.00528]] Large Language Models for Simultaneous Named Entity Extraction and Spelling Correction(https://arxiv.org/abs/2403.00528)
• Keywords: extraction, generative, large language model
• Abstract: Language Models (LMs) such as BERT, have been shown to perform well on the task of identifying Named Entities (NE) in text. A BERT LM is typically used as a classifier to classify individual tokens in the input text, or to classify spans of tokens, as belonging to one of a set of possible NE categories. In this paper, we hypothesise that decoder-only Large Language Models (LLMs) can also be used generatively to extract both the NE, as well as potentially recover the correct surface form of the NE, where any spelling errors that were present in the input text get automatically corrected. We fine-tune two BERT LMs as baselines, as well as eight open-source LLMs, on the task of producing NEs from text that was obtained by applying Optical Character Recognition (OCR) to images of Japanese shop receipts; in this work, we do not attempt to find or evaluate the location of NEs in the text. We show that the best fine-tuned LLM performs as well as, or slightly better than, the best fine-tuned BERT LM, although the differences are not significant. However, the best LLM is also shown to correct OCR errors in some cases, as initially hypothesised.

Title: SURE: SUrvey REcipes for building reliable and robust deep networks

• Authors: Yuting Li, Yingyi Chen, Xuanlong Yu, Dexiong Chen, Xi Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00543
• Pdf URL: https://arxiv.org/pdf/2403.00543
• Copy Paste: [[2403.00543]] SURE: SUrvey REcipes for building reliable and robust deep networks(https://arxiv.org/abs/2403.00543)
• Keywords: robust
• Abstract: In this paper, we revisit techniques for uncertainty estimation within deep neural networks and consolidate a suite of techniques to enhance their reliability. Our investigation reveals that an integrated application of diverse techniques--spanning model regularization, classifier and optimization--substantially improves the accuracy of uncertainty predictions in image classification tasks. The synergistic effect of these techniques culminates in our novel SURE approach. We rigorously evaluate SURE against the benchmark of failure prediction, a critical testbed for uncertainty estimation efficacy. Our results showcase a consistently better performance than models that individually deploy each technique, across various datasets and model architectures. When applied to real-world challenges, such as data corruption, label noise, and long-tailed class distribution, SURE exhibits remarkable robustness, delivering results that are superior or on par with current state-of-the-art specialized methods. Particularly on Animal-10N and Food-101N for learning with noisy labels, SURE achieves state-of-the-art performance without any task-specific adjustments. This work not only sets a new benchmark for robust uncertainty estimation but also paves the way for its application in diverse, real-world scenarios where reliability is paramount. Our code is available at \url{https://yutingli0606.github.io/SURE/}.

Title: Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores

• Authors: Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F. Siu, Byron C. Wallace, Ani Nenkova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00553
• Pdf URL: https://arxiv.org/pdf/2403.00553
• Copy Paste: [[2403.00553]] Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores(https://arxiv.org/abs/2403.00553)
• Keywords: generative, large language model
• Abstract: The diversity across outputs generated by large language models shapes the perception of their quality and utility. Prompt leaks, templated answer structure, and canned responses across different interactions are readily noticed by people, but there is no standard score to measure this aspect of model behavior. In this work we empirically investigate diversity scores on English texts. We find that computationally efficient compression algorithms capture information similar to what is measured by slow to compute $n$-gram overlap homogeneity scores. Further, a combination of measures -- compression ratios, self-repetition of long $n$-grams and Self-BLEU and BERTScore -- are sufficient to report, as they have low mutual correlation with each other. The applicability of scores extends beyond analysis of generative models; for example, we highlight applications on instruction-tuning datasets and human-produced texts. We release a diversity score package to facilitate research and invite consistency across reports.

Title: Lincoln's Annotated Spatio-Temporal Strawberry Dataset (LAST-Straw)

• Authors: Katherine Margaret Frances James, Karoline Heiwolt, Daniel James Sargent, Grzegorz Cielniak
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00566
• Pdf URL: https://arxiv.org/pdf/2403.00566
• Copy Paste: [[2403.00566]] Lincoln's Annotated Spatio-Temporal Strawberry Dataset (LAST-Straw)(https://arxiv.org/abs/2403.00566)
• Keywords: extraction, segmentation
• Abstract: Automated phenotyping of plants for breeding and plant studies promises to provide quantitative metrics on plant traits at a previously unattainable observation frequency. Developers of tools for performing high-throughput phenotyping are, however, constrained by the availability of relevant datasets on which to perform validation. To this end, we present a spatio-temporal dataset of 3D point clouds of strawberry plants for two varieties, totalling 84 individual point clouds. We focus on the end use of such tools - the extraction of biologically relevant phenotypes - and demonstrate a phenotyping pipeline on the dataset. This comprises of the steps, including; segmentation, skeletonisation and tracking, and we detail how each stage facilitates the extraction of different phenotypes or provision of data insights. We particularly note that assessment is focused on the validation of phenotypes, extracted from the representations acquired at each step of the pipeline, rather than singularly focusing on assessing the representation itself. Therefore, where possible, we provide \textit{in silico} ground truth baselines for the phenotypes extracted at each step and introduce methodology for the quantitative assessment of skeletonisation and the length trait extracted thereof. This dataset contributes to the corpus of freely available agricultural/horticultural spatio-temporal data for the development of next-generation phenotyping tools, increasing the number of plant varieties available for research in this field and providing a basis for genuine comparison of new phenotyping methodology.

Title: Rethinking cluster-conditioned diffusion models

• Authors: Nikolas Adaloglou, Tim Kaiser, Felix Michels, Markus Kollmann
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00570
• Pdf URL: https://arxiv.org/pdf/2403.00570
• Copy Paste: [[2403.00570]] Rethinking cluster-conditioned diffusion models(https://arxiv.org/abs/2403.00570)
• Keywords: diffusion
• Abstract: We present a comprehensive experimental study on image-level conditioning for diffusion models using cluster assignments. We elucidate how individual components regarding image clustering impact image synthesis across three datasets. By combining recent advancements from image clustering and diffusion models, we show that, given the optimal cluster granularity with respect to image synthesis (visual groups), cluster-conditioning can achieve state-of-the-art FID (i.e. 1.67, 2.17 on CIFAR10 and CIFAR100 respectively), while attaining a strong training sample efficiency. Finally, we propose a novel method to derive an upper cluster bound that reduces the search space of the visual groups using solely feature-based clustering. Unlike existing approaches, we find no significant connection between clustering and cluster-conditional image generation. The code and cluster assignments will be released.

Title: Beyond Single-Model Views for Deep Learning: Optimization versus Generalizability of Stochastic Optimization Algorithms

• Authors: Toki Tahmid Inan, Mingrui Liu, Amarda Shehu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00574
• Pdf URL: https://arxiv.org/pdf/2403.00574
• Copy Paste: [[2403.00574]] Beyond Single-Model Views for Deep Learning: Optimization versus Generalizability of Stochastic Optimization Algorithms(https://arxiv.org/abs/2403.00574)
• Keywords: fair
• Abstract: Despite an extensive body of literature on deep learning optimization, our current understanding of what makes an optimization algorithm effective is fragmented. In particular, we do not understand well whether enhanced optimization translates to improved generalizability. Current research overlooks the inherent stochastic nature of stochastic gradient descent (SGD) and its variants, resulting in a lack of comprehensive benchmarking and insight into their statistical performance. This paper aims to address this gap by adopting a novel approach. Rather than solely evaluating the endpoint of individual optimization trajectories, we draw from an ensemble of trajectories to estimate the stationary distribution of stochastic optimizers. Our investigation encompasses a wide array of techniques, including SGD and its variants, flat-minima optimizers, and new algorithms we propose under the Basin Hopping framework. Through our evaluation, which encompasses synthetic functions with known minima and real-world problems in computer vision and natural language processing, we emphasize fair benchmarking under a statistical framework, comparing stationary distributions and establishing statistical significance. Our study uncovers several key findings regarding the relationship between training loss and hold-out accuracy, as well as the comparable performance of SGD, noise-enabled variants, and novel optimizers utilizing the BH framework. Notably, these algorithms demonstrate performance on par with flat-minima optimizers like SAM, albeit with half the gradient evaluations. We anticipate that our work will catalyze further exploration in deep learning optimization, encouraging a shift away from single-model approaches towards methodologies that acknowledge and leverage the stochastic nature of optimizers.

Title: Improving Explicit Spatial Relationships in Text-to-Image Generation through an Automatically Derived Dataset

• Authors: Ander Salaberria, Gorka Azkune, Oier Lopez de Lacalle, Aitor Soroa, Eneko Agirre, Frank Keller
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00587
• Pdf URL: https://arxiv.org/pdf/2403.00587
• Copy Paste: [[2403.00587]] Improving Explicit Spatial Relationships in Text-to-Image Generation through an Automatically Derived Dataset(https://arxiv.org/abs/2403.00587)
• Keywords: diffusion
• Abstract: Existing work has observed that current text-to-image systems do not accurately reflect explicit spatial relations between objects such as 'left of' or 'below'. We hypothesize that this is because explicit spatial relations rarely appear in the image captions used to train these models. We propose an automatic method that, given existing images, generates synthetic captions that contain 14 explicit spatial relations. We introduce the Spatial Relation for Generation (SR4G) dataset, which contains 9.9 millions image-caption pairs for training, and more than 60 thousand captions for evaluation. In order to test generalization we also provide an 'unseen' split, where the set of objects in the train and test captions are disjoint. SR4G is the first dataset that can be used to spatially fine-tune text-to-image systems. We show that fine-tuning two different Stable Diffusion models (denoted as SD$_{SR4G}$) yields up to 9 points improvements in the VISOR metric. The improvement holds in the 'unseen' split, showing that SD$_{SR4G}$ is able to generalize to unseen objects. SD$_{SR4G}$ improves the state-of-the-art with fewer parameters, and avoids complex architectures. Our analysis shows that improvement is consistent for all relations. The dataset and the code will be publicly available.

Title: Rethinking Few-shot 3D Point Cloud Semantic Segmentation

• Authors: Zhaochong An, Guolei Sun, Yun Liu, Fayao Liu, Zongwei Wu, Dan Wang, Luc Van Gool, Serge Belongie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00592
• Pdf URL: https://arxiv.org/pdf/2403.00592
• Copy Paste: [[2403.00592]] Rethinking Few-shot 3D Point Cloud Semantic Segmentation(https://arxiv.org/abs/2403.00592)
• Keywords: segmentation
• Abstract: This paper revisits few-shot 3D point cloud semantic segmentation (FS-PCS), with a focus on two significant issues in the state-of-the-art: foreground leakage and sparse point distribution. The former arises from non-uniform point sampling, allowing models to distinguish the density disparities between foreground and background for easier segmentation. The latter results from sampling only 2,048 points, limiting semantic information and deviating from the real-world practice. To address these issues, we introduce a standardized FS-PCS setting, upon which a new benchmark is built. Moreover, we propose a novel FS-PCS model. While previous methods are based on feature optimization by mainly refining support features to enhance prototypes, our method is based on correlation optimization, referred to as Correlation Optimization Segmentation (COSeg). Specifically, we compute Class-specific Multi-prototypical Correlation (CMC) for each query point, representing its correlations to category prototypes. Then, we propose the Hyper Correlation Augmentation (HCA) module to enhance CMC. Furthermore, tackling the inherent property of few-shot training to incur base susceptibility for models, we propose to learn non-parametric prototypes for the base classes during training. The learned base prototypes are used to calibrate correlations for the background class through a Base Prototypes Calibration (BPC) module. Experiments on popular datasets demonstrate the superiority of COSeg over existing methods. The code is available at: https://github.com/ZhaochongAn/COSeg

Title: Flattening Singular Values of Factorized Convolution for Medical Images

• Authors: Zexin Feng, Na Zeng, Jiansheng Fang, Xingyue Wang, Xiaoxi Lu, Heng Meng, Jiang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00606
• Pdf URL: https://arxiv.org/pdf/2403.00606
• Copy Paste: [[2403.00606]] Flattening Singular Values of Factorized Convolution for Medical Images(https://arxiv.org/abs/2403.00606)
• Keywords: robust
• Abstract: Convolutional neural networks (CNNs) have long been the paradigm of choice for robust medical image processing (MIP). Therefore, it is crucial to effectively and efficiently deploy CNNs on devices with different computing capabilities to support computer-aided diagnosis. Many methods employ factorized convolutional layers to alleviate the burden of limited computational resources at the expense of expressiveness. To this end, given weak medical image-driven CNN model optimization, a Singular value equalization generalizer-induced Factorized Convolution (SFConv) is proposed to improve the expressive power of factorized convolutions in MIP models. We first decompose the weight matrix of convolutional filters into two low-rank matrices to achieve model reduction. Then minimize the KL divergence between the two low-rank weight matrices and the uniform distribution, thereby reducing the number of singular value directions with significant variance. Extensive experiments on fundus and OCTA datasets demonstrate that our SFConv yields competitive expressiveness over vanilla convolutions while reducing complexity.

Title: Bias Mitigation in Fine-tuning Pre-trained Models for Enhanced Fairness and Efficiency

• Authors: Yixuan Zhang, Feng Zhou
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.00625
• Pdf URL: https://arxiv.org/pdf/2403.00625
• Copy Paste: [[2403.00625]] Bias Mitigation in Fine-tuning Pre-trained Models for Enhanced Fairness and Efficiency(https://arxiv.org/abs/2403.00625)
• Keywords: robust, fair
• Abstract: Fine-tuning pre-trained models is a widely employed technique in numerous real-world applications. However, fine-tuning these models on new tasks can lead to unfair outcomes. This is due to the absence of generalization guarantees for fairness properties, regardless of whether the original pre-trained model was developed with fairness considerations. To tackle this issue, we introduce an efficient and robust fine-tuning framework specifically designed to mitigate biases in new tasks. Our empirical analysis shows that the parameters in the pre-trained model that affect predictions for different demographic groups are different, so based on this observation, we employ a transfer learning strategy that neutralizes the importance of these influential weights, determined using Fisher information across demographic groups. Additionally, we integrate this weight importance neutralization strategy with a matrix factorization technique, which provides a low-rank approximation of the weight matrix using fewer parameters, reducing the computational demands. Experiments on multiple pre-trained models and new tasks demonstrate the effectiveness of our method.

Title: Region-Adaptive Transform with Segmentation Prior for Image Compression

• Authors: Yuxi Liu, Wenhan Yang, Huihui Bai, Yunchao Wei, Yao Zhao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.00628
• Pdf URL: https://arxiv.org/pdf/2403.00628
• Copy Paste: [[2403.00628]] Region-Adaptive Transform with Segmentation Prior for Image Compression(https://arxiv.org/abs/2403.00628)
• Keywords: segmentation
• Abstract: Learned Image Compression (LIC) has shown remarkable progress in recent years. Existing works commonly employ CNN-based or self-attention-based modules as transform methods for compression. However, there is no prior research on neural transform that focuses on specific regions. In response, we introduce the class-agnostic segmentation masks (i.e. semantic masks without category labels) for extracting region-adaptive contextual information. Our proposed module, Region-Adaptive Transform, applies adaptive convolutions on different regions guided by the masks. Additionally, we introduce a plug-and-play module named Scale Affine Layer to incorporate rich contexts from various regions. While there have been prior image compression efforts that involve segmentation masks as additional intermediate inputs, our approach differs significantly from them. Our advantages lie in that, to avoid extra bitrate overhead, we treat these masks as privilege information, which is accessible during the model training stage but not required during the inference phase. To the best of our knowledge, we are the first to employ class-agnostic masks as privilege information and achieve superior performance in pixel-fidelity metrics, such as Peak Signal to Noise Ratio (PSNR). The experimental results demonstrate our improvement compared to previously well-performing methods, with about 8.2% bitrate saving compared to VTM-17.0. The code will be released at https://github.com/GityuxiLiu/Region-Adaptive-Transform-with-Segmentation-Prior-for-Image-Compression.

Title: Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks

• Authors: Yuhao Liu, Fang Liu, Zhanghan Ke, Nanxuan Zhao, Rynson W.H. Lau
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00644
• Pdf URL: https://arxiv.org/pdf/2403.00644
• Copy Paste: [[2403.00644]] Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks(https://arxiv.org/abs/2403.00644)
• Keywords: robust, diffusion
• Abstract: Diffusion models trained on large-scale datasets have achieved remarkable progress in image synthesis. However, due to the randomness in the diffusion process, they often struggle with handling diverse low-level tasks that require details preservation. To overcome this limitation, we present a new Diff-Plugin framework to enable a single pre-trained diffusion model to generate high-fidelity results across a variety of low-level tasks. Specifically, we first propose a lightweight Task-Plugin module with a dual branch design to provide task-specific priors, guiding the diffusion process in preserving image content. We then propose a Plugin-Selector that can automatically select different Task-Plugins based on the text instruction, allowing users to edit images by indicating multiple low-level tasks with natural language. We conduct extensive experiments on 8 low-level vision tasks. The results demonstrate the superiority of Diff-Plugin over existing methods, particularly in real-world scenarios. Our ablations further validate that Diff-Plugin is stable, schedulable, and supports robust training across different dataset sizes.

Title: COLON: The largest COlonoscopy LONg sequence public database

• Authors: Lina Ruiz, Franklin Sierra-Jerez, Jair Ruiz, Fabio Martinez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00663
• Pdf URL: https://arxiv.org/pdf/2403.00663
• Copy Paste: [[2403.00663]] COLON: The largest COlonoscopy LONg sequence public database(https://arxiv.org/abs/2403.00663)
• Keywords: robust, segmentation
• Abstract: Colorectal cancer is the third most aggressive cancer worldwide. Polyps, as the main biomarker of the disease, are detected, localized, and characterized through colonoscopy procedures. Nonetheless, during the examination, up to 25% of polyps are missed, because of challenging conditions (camera movements, lighting changes), and the close similarity of polyps and intestinal folds. Besides, there is a remarked subjectivity and expert dependency to observe and detect abnormal regions along the intestinal tract. Currently, publicly available polyp datasets have allowed significant advances in computational strategies dedicated to characterizing non-parametric polyp shapes. These computational strategies have achieved remarkable scores of up to 90% in segmentation tasks. Nonetheless, these strategies operate on cropped and expert-selected frames that always observe polyps. In consequence, these computational approximations are far from clinical scenarios and real applications, where colonoscopies are redundant on intestinal background with high textural variability. In fact, the polyps typically represent less than 1% of total observations in a complete colonoscopy record. This work introduces COLON: the largest COlonoscopy LONg sequence dataset with around of 30 thousand polyp labeled frames and 400 thousand background frames. The dataset was collected from a total of 30 complete colonoscopies with polyps at different stages, variations in preparation procedures, and some cases the observation of surgical instrumentation. Additionally, 10 full intestinal background video control colonoscopies were integrated in order to achieve a robust polyp-background frame differentiation. The COLON dataset is open to the scientific community to bring new scenarios to propose computational tools dedicated to polyp detection and segmentation over long sequences, being closer to real colonoscopy scenarios.

Title: Advancing Additive Manufacturing through Deep Learning: A Comprehensive Review of Current Progress and Future Challenges

• Authors: Amirul Islam Saimon, Emmanuel Yangue, Xiaowei Yue, Zhenyu (James)Kong, Chenang Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00669
• Pdf URL: https://arxiv.org/pdf/2403.00669
• Copy Paste: [[2403.00669]] Advancing Additive Manufacturing through Deep Learning: A Comprehensive Review of Current Progress and Future Challenges(https://arxiv.org/abs/2403.00669)
• Keywords: extraction, generative
• Abstract: Additive manufacturing (AM) has already proved itself to be the potential alternative to widely-used subtractive manufacturing due to its extraordinary capacity of manufacturing highly customized products with minimum material wastage. Nevertheless, it is still not being considered as the primary choice for the industry due to some of its major inherent challenges, including complex and dynamic process interactions, which are sometimes difficult to fully understand even with traditional machine learning because of the involvement of high-dimensional data such as images, point clouds, and voxels. However, the recent emergence of deep learning (DL) is showing great promise in overcoming many of these challenges as DL can automatically capture complex relationships from high-dimensional data without hand-crafted feature extraction. Therefore, the volume of research in the intersection of AM and DL is exponentially growing each year which makes it difficult for the researchers to keep track of the trend and future potential directions. Furthermore, to the best of our knowledge, there is no comprehensive review paper in this research track summarizing the recent studies. Therefore, this paper reviews the recent studies that apply DL for making the AM process better with a high-level summary of their contributions and limitations. Finally, it summarizes the current challenges and recommends some of the promising opportunities in this domain for further investigation with a special focus on generalizing DL models for wide-range of geometry types, managing uncertainties both in AM data and DL models, overcoming limited and noisy AM data issues by incorporating generative models, and unveiling the potential of interpretable DL for AM.

Title: Self-Consistent Decoding for More Factual Open Responses

• Authors: Christopher Malon, Xiaodan Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00696
• Pdf URL: https://arxiv.org/pdf/2403.00696
• Copy Paste: [[2403.00696]] Self-Consistent Decoding for More Factual Open Responses(https://arxiv.org/abs/2403.00696)
• Keywords: large language model
• Abstract: Self-consistency has emerged as a powerful method for improving the accuracy of short answers generated by large language models. As previously defined, it only concerns the accuracy of a final answer parsed from generated text. In this work, we extend the idea to open response generation, by integrating voting into the decoding method. Each output sentence is selected from among multiple samples, conditioning on the previous selections, based on a simple token overlap score. We compare this "Sample & Select" method to greedy decoding, beam search, nucleus sampling, and the recently introduced hallucination avoiding decoders of DoLA, P-CRR, and S-CRR. We show that Sample & Select improves factuality by a 30% relative margin against these decoders in NLI-based evaluation on the subsets of CNN/DM and XSum used in the FRANK benchmark, while maintaining comparable ROUGE-1 F1 scores against reference summaries. We collect human verifications of the generated summaries, confirming the factual superiority of our method.

Title: Few-Shot Relation Extraction with Hybrid Visual Evidence

• Authors: Jiaying Gong, Hoda Eldardiry
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.00724
• Pdf URL: https://arxiv.org/pdf/2403.00724
• Copy Paste: [[2403.00724]] Few-Shot Relation Extraction with Hybrid Visual Evidence(https://arxiv.org/abs/2403.00724)
• Keywords: extraction
• Abstract: The goal of few-shot relation extraction is to predict relations between name entities in a sentence when only a few labeled instances are available for training. Existing few-shot relation extraction methods focus on uni-modal information such as text only. This reduces performance when there are no clear contexts between the name entities described in text. We propose a multi-modal few-shot relation extraction model (MFS-HVE) that leverages both textual and visual semantic information to learn a multi-modal representation jointly. The MFS-HVE includes semantic feature extractors and multi-modal fusion components. The MFS-HVE semantic feature extractors are developed to extract both textual and visual features. The visual features include global image features and local object features within the image. The MFS-HVE multi-modal fusion unit integrates information from various modalities using image-guided attention, object-guided attention, and hybrid feature attention to fully capture the semantic interaction between visual regions of images and relevant texts. Extensive experiments conducted on two public datasets demonstrate that semantic visual information significantly improves the performance of few-shot relation prediction.

Title: Can Transformers Capture Spatial Relations between Objects?

• Authors: Chuan Wen, Dinesh Jayaraman, Yang Gao
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.00729
• Pdf URL: https://arxiv.org/pdf/2403.00729
• Copy Paste: [[2403.00729]] Can Transformers Capture Spatial Relations between Objects?(https://arxiv.org/abs/2403.00729)
• Keywords: transformer
• Abstract: Spatial relationships between objects represent key scene information for humans to understand and interact with the world. To study the capability of current computer vision systems to recognize physically grounded spatial relations, we start by proposing precise relation definitions that permit consistently annotating a benchmark dataset. Despite the apparent simplicity of this task relative to others in the recognition literature, we observe that existing approaches perform poorly on this benchmark. We propose new approaches exploiting the long-range attention capabilities of transformers for this task, and evaluating key design principles. We identify a simple "RelatiViT" architecture and demonstrate that it outperforms all current approaches. To our knowledge, this is the first method to convincingly outperform naive baselines on spatial relation prediction in in-the-wild settings. The code and datasets are available in \url{https://sites.google.com/view/spatial-relation}.

Title: Dialect prejudice predicts AI decisions about people's character, employability, and criminality

• Authors: Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, Sharese King
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.00742
• Pdf URL: https://arxiv.org/pdf/2403.00742
• Copy Paste: [[2403.00742]] Dialect prejudice predicts AI decisions about people's character, employability, and criminality(https://arxiv.org/abs/2403.00742)
• Keywords: fair
• Abstract: Hundreds of millions of people now interact with language models, with uses ranging from serving as a writing aid to informing hiring decisions. Yet these language models are known to perpetuate systematic racial prejudices, making their judgments biased in problematic ways about groups like African Americans. While prior research has focused on overt racism in language models, social scientists have argued that racism with a more subtle character has developed over time. It is unknown whether this covert racism manifests in language models. Here, we demonstrate that language models embody covert racism in the form of dialect prejudice: we extend research showing that Americans hold raciolinguistic stereotypes about speakers of African American English and find that language models have the same prejudice, exhibiting covert stereotypes that are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to the ones from before the civil rights movement. By contrast, the language models' overt stereotypes about African Americans are much more positive. We demonstrate that dialect prejudice has the potential for harmful consequences by asking language models to make hypothetical decisions about people, based only on how they speak. Language models are more likely to suggest that speakers of African American English be assigned less prestigious jobs, be convicted of crimes, and be sentenced to death. Finally, we show that existing methods for alleviating racial bias in language models such as human feedback training do not mitigate the dialect prejudice, but can exacerbate the discrepancy between covert and overt stereotypes, by teaching language models to superficially conceal the racism that they maintain on a deeper level. Our findings have far-reaching implications for the fair and safe employment of language technology.

Title: AtP*: An efficient and scalable method for localizing LLM behaviour to components

• Authors: János Kramár, Tom Lieberum, Rohin Shah, Neel Nanda (Google DeepMind)
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.00745
• Pdf URL: https://arxiv.org/pdf/2403.00745
• Copy Paste: [[2403.00745]] AtP*: An efficient and scalable method for localizing LLM behaviour to components(https://arxiv.org/abs/2403.00745)
• Keywords: large language model
• Abstract: Activation Patching is a method of directly computing causal attributions of behavior to model components. However, applying it exhaustively requires a sweep with cost scaling linearly in the number of model components, which can be prohibitively expensive for SoTA Large Language Models (LLMs). We investigate Attribution Patching (AtP), a fast gradient-based approximation to Activation Patching and find two classes of failure modes of AtP which lead to significant false negatives. We propose a variant of AtP called AtP*, with two changes to address these failure modes while retaining scalability. We present the first systematic study of AtP and alternative methods for faster activation patching and show that AtP significantly outperforms all other investigated methods, with AtP* providing further significant improvement. Finally, we provide a method to bound the probability of remaining false negatives of AtP* estimates.

Title: Mitigating Reversal Curse via Semantic-aware Permutation Training

• Authors: Qingyan Guo, Rui Wang, Junliang Guo, Xu Tan, Jiang Bian, Yujiu Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00758
• Pdf URL: https://arxiv.org/pdf/2403.00758
• Copy Paste: [[2403.00758]] Mitigating Reversal Curse via Semantic-aware Permutation Training(https://arxiv.org/abs/2403.00758)
• Keywords: large language model
• Abstract: While large language models (LLMs) have achieved impressive performance across diverse tasks, recent studies showcase that causal LLMs suffer from the "reversal curse". It is a typical example that the model knows "A's father is B", but is unable to reason "B's child is A". This limitation poses a challenge to the advancement of artificial general intelligence (AGI), as it suggests a gap in the models' ability to comprehend and apply bidirectional reasoning. In this paper, we first conduct substantial evaluation and identify that the root cause of the reversal curse lies in the different word order between the training and inference stage, namely, the poor ability of causal language models to predict antecedent words within the training data. Accordingly, permutation on the training data is considered as a potential solution, since this can make the model predict antecedent words or tokens. However, previous permutation methods may disrupt complete phrases or entities, thereby posing challenges for the model to comprehend and learn from training data. To address this issue, we propose Semantic-aware Permutation Training (SPT), which addresses this issue by segmenting the training sentences into semantic units (i.e., entities or phrases) with an assistant language model and permuting these units before feeding into the model. Extensive experiments demonstrate that SPT effectively mitigates the reversal curse since the performance on reversed questions approximates that on the forward ones, and significantly advances the performance of existing works.